[MarkLogic Dev General] CPF pre-commit action?

Geert Josten Geert.Josten at marklogic.com
Mon Jan 18 11:37:54 PST 2016


Hi Will,

It doesn¹t have to be that much more complicated. Just add a ¹staging¹
database, put your CPF there, and insert docs into that. Add one extra
state and pipeline action at the end of your chain that copies the final
docs into a different database, duplicating any properties..

It would become slightly more complicated if there are linked docs that
needs to be copied as well, but as long as you keep db uris the same, you
don¹t need to touch the contents of any of the docs..

Kind regards,
Geert

On 1/18/16, 6:33 PM, "general-bounces at developer.marklogic.com on behalf of
Will Thompson" <general-bounces at developer.marklogic.com on behalf of
wthompson at oconnors.com> wrote:

>Geert,
>
>I hadn't considered that, but it would definitely achieve the desired
>behavior. Thanks for the suggestion. It would add a lot more complexity
>to the pipeline, but its sounds feasible.
>
>-Will
>
>> On Jan 15, 2016, at 1:38 PM, Geert Josten <Geert.Josten at marklogic.com>
>>wrote:
>> 
>> Have you considered taking an approach similar to Info Studio: insert
>> unprocessed docs in an separate database in which you have all the CPF
>> pipelines you need. Not until finishing of processing it gets copied
>>into
>> the target database. Info Studio was using the Fab database for that
>> purpose..
>> 
>> Cheers
>> 
>> On 1/15/16, 5:06 PM, "general-bounces at developer.marklogic.com on behalf
>>of
>> Will Thompson" <general-bounces at developer.marklogic.com on behalf of
>> wthompson at oconnors.com> wrote:
>> 
>>> Geert,
>>> 
>>> We're using CPF because some steps may necessitate human intervention,
>>>in
>>> which case they go into a queue, get resolved, a state change occurs,
>>>and
>>> the pipeline marches on. But are you suggesting a pre-commit trigger to
>>> supplement the existing CPF pipeline?
>>> 
>>> Danny,
>>> 
>>> The main issue for us is user experience. For example, in most cases
>>> document X will already exist in the database and users can see it as
>>> part of the application. We want to replace document X with an updated
>>> version and not have it appear visible to users until it has been
>>> transformed. Hiding it in a collection would hide it from users as
>>>well.
>>> I said "version" so maybe DLS seems like a good fit? But I'm still
>>>unsure
>>> of how to begin the pipeline with the document in a checked-out state.
>>> 
>>> We do have a solution to this, which is essentially: the application
>>> takes care of it. But I wanted to see if it were possible to push that
>>> logic back into CPF and simplify our API such that other applications
>>> could utilize the pipeline with basic insert and delete operations.
>>> 
>>> Thanks,
>>> 
>>> Will
>>> 
>>> 
>>>> On Jan 15, 2016, at 5:26 AM, Geert Josten <Geert.Josten at marklogic.com>
>>>> wrote:
>>>> 
>>>> Or how about KISS: just use one pre-commit trigger that does all the
>>>> transformation/processing in one blow..
>>>> 
>>>> Kind regards,
>>>> Geert
>>>> 
>>>> On 1/15/16, 12:42 AM, "general-bounces at developer.marklogic.com on
>>>>behalf
>>>> of Danny Sokolsky" <general-bounces at developer.marklogic.com on behalf
>>>>of
>>>> Danny.Sokolsky at marklogic.com> wrote:
>>>> 
>>>>> Hi Will,
>>>>> 
>>>>> I can't think of a way to do this without modifying the core cpf
>>>>>code.
>>>>> For some of the status-change-handling pipeline steps, there are
>>>>> pre-commit triggers, but those are for cpf bookkeeping mostly, I
>>>>> believe.
>>>>> And I would not recommend changing that code.
>>>>> 
>>>>> But I would question why you would want to do this.  CPF is designed
>>>>>to
>>>>> be resilient and to allow you to have multiple steps in your
>>>>>pipeline.
>>>>> Why not make a step that does the transform its own step?  If you do
>>>>> not
>>>>> want the document to be visible until it is transformed, then put it
>>>>>in
>>>>> some collection initially that is not seen by your application, and
>>>>> then
>>>>> take it out of that collection when you are done with the transform
>>>>>(or
>>>>> something similar to that).
>>>>> 
>>>>> Is it just because you want to have one less step in your pipeline
>>>>>that
>>>>> you want to do this?
>>>>> 
>>>>> -Danny
>>>>> 
>>>>> -----Original Message-----
>>>>> From: general-bounces at developer.marklogic.com
>>>>> [mailto:general-bounces at developer.marklogic.com] On Behalf Of Will
>>>>> Thompson
>>>>> Sent: Thursday, January 14, 2016 1:06 PM
>>>>> To: MarkLogic Developer Discussion
>>>>> Subject: [MarkLogic Dev General] CPF pre-commit action?
>>>>> 
>>>>> Is it possible to configure a CPF pipeline such that when a document
>>>>>is
>>>>> inserted, transformations are first executed on the document in a
>>>>> pre-commit stage, and if those complete successfully, then the
>>>>> transformation result is what's finally committed at that URI? And
>>>>>any
>>>>> remaining pipeline could carry on without the pre-transformed data
>>>>>ever
>>>>> being visible to the database?
>>>>> 
>>>>> -Will
>>>>> _______________________________________________
>>>>> General mailing list
>>>>> General at developer.marklogic.com
>>>>> Manage your subscription at:
>>>>> http://developer.marklogic.com/mailman/listinfo/general
>>>>> _______________________________________________
>>>>> General mailing list
>>>>> General at developer.marklogic.com
>>>>> Manage your subscription at:
>>>>> http://developer.marklogic.com/mailman/listinfo/general
>>>> 
>>>> _______________________________________________
>>>> General mailing list
>>>> General at developer.marklogic.com
>>>> Manage your subscription at:
>>>> http://developer.marklogic.com/mailman/listinfo/general
>>> 
>>> _______________________________________________
>>> General mailing list
>>> General at developer.marklogic.com
>>> Manage your subscription at:
>>> http://developer.marklogic.com/mailman/listinfo/general
>> 
>> _______________________________________________
>> General mailing list
>> General at developer.marklogic.com
>> Manage your subscription at:
>> http://developer.marklogic.com/mailman/listinfo/general
>
>_______________________________________________
>General mailing list
>General at developer.marklogic.com
>Manage your subscription at:
>http://developer.marklogic.com/mailman/listinfo/general



More information about the General mailing list