[MarkLogic Dev General] CPF pre-commit action?

Danny Sokolsky Danny.Sokolsky at marklogic.com
Mon Jan 18 11:46:04 PST 2016


And if your properties are only being used by cpf, then the last step can skip moving over the properties to the prod database, thus saving you those fragments in that database.

-Danny

-----Original Message-----
From: general-bounces at developer.marklogic.com [mailto:general-bounces at developer.marklogic.com] On Behalf Of Geert Josten
Sent: Monday, January 18, 2016 11:38 AM
To: MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] CPF pre-commit action?

Hi Will,

It doesn¹t have to be that much more complicated. Just add a ¹staging¹ database, put your CPF there, and insert docs into that. Add one extra state and pipeline action at the end of your chain that copies the final docs into a different database, duplicating any properties..

It would become slightly more complicated if there are linked docs that needs to be copied as well, but as long as you keep db uris the same, you don¹t need to touch the contents of any of the docs..

Kind regards,
Geert

On 1/18/16, 6:33 PM, "general-bounces at developer.marklogic.com on behalf of Will Thompson" <general-bounces at developer.marklogic.com on behalf of wthompson at oconnors.com> wrote:

>Geert,
>
>I hadn't considered that, but it would definitely achieve the desired 
>behavior. Thanks for the suggestion. It would add a lot more complexity 
>to the pipeline, but its sounds feasible.
>
>-Will
>
>> On Jan 15, 2016, at 1:38 PM, Geert Josten 
>><Geert.Josten at marklogic.com>
>>wrote:
>> 
>> Have you considered taking an approach similar to Info Studio: insert  
>>unprocessed docs in an separate database in which you have all the CPF  
>>pipelines you need. Not until finishing of processing it gets copied 
>>into  the target database. Info Studio was using the Fab database for 
>>that  purpose..
>> 
>> Cheers
>> 
>> On 1/15/16, 5:06 PM, "general-bounces at developer.marklogic.com on 
>>behalf of  Will Thompson" <general-bounces at developer.marklogic.com on 
>>behalf of  wthompson at oconnors.com> wrote:
>> 
>>> Geert,
>>> 
>>> We're using CPF because some steps may necessitate human 
>>>intervention, in  which case they go into a queue, get resolved, a 
>>>state change occurs, and  the pipeline marches on. But are you 
>>>suggesting a pre-commit trigger to  supplement the existing CPF 
>>>pipeline?
>>> 
>>> Danny,
>>> 
>>> The main issue for us is user experience. For example, in most cases  
>>>document X will already exist in the database and users can see it as  
>>>part of the application. We want to replace document X with an 
>>>updated  version and not have it appear visible to users until it has 
>>>been  transformed. Hiding it in a collection would hide it from users 
>>>as well.
>>> I said "version" so maybe DLS seems like a good fit? But I'm still 
>>>unsure  of how to begin the pipeline with the document in a 
>>>checked-out state.
>>> 
>>> We do have a solution to this, which is essentially: the application 
>>> takes care of it. But I wanted to see if it were possible to push 
>>> that logic back into CPF and simplify our API such that other 
>>> applications could utilize the pipeline with basic insert and delete operations.
>>> 
>>> Thanks,
>>> 
>>> Will
>>> 
>>> 
>>>> On Jan 15, 2016, at 5:26 AM, Geert Josten 
>>>> <Geert.Josten at marklogic.com>
>>>> wrote:
>>>> 
>>>> Or how about KISS: just use one pre-commit trigger that does all 
>>>> the transformation/processing in one blow..
>>>> 
>>>> Kind regards,
>>>> Geert
>>>> 
>>>> On 1/15/16, 12:42 AM, "general-bounces at developer.marklogic.com on 
>>>>behalf  of Danny Sokolsky" <general-bounces at developer.marklogic.com 
>>>>on behalf of  Danny.Sokolsky at marklogic.com> wrote:
>>>> 
>>>>> Hi Will,
>>>>> 
>>>>> I can't think of a way to do this without modifying the core cpf 
>>>>>code.
>>>>> For some of the status-change-handling pipeline steps, there are  
>>>>>pre-commit triggers, but those are for cpf bookkeeping mostly, I  
>>>>>believe.
>>>>> And I would not recommend changing that code.
>>>>> 
>>>>> But I would question why you would want to do this.  CPF is 
>>>>>designed to  be resilient and to allow you to have multiple steps 
>>>>>in your pipeline.
>>>>> Why not make a step that does the transform its own step?  If you 
>>>>>do  not  want the document to be visible until it is transformed, 
>>>>>then put it in  some collection initially that is not seen by your 
>>>>>application, and  then  take it out of that collection when you are 
>>>>>done with the transform (or  something similar to that).
>>>>> 
>>>>> Is it just because you want to have one less step in your pipeline 
>>>>>that  you want to do this?
>>>>> 
>>>>> -Danny
>>>>> 
>>>>> -----Original Message-----
>>>>> From: general-bounces at developer.marklogic.com
>>>>> [mailto:general-bounces at developer.marklogic.com] On Behalf Of Will 
>>>>> Thompson
>>>>> Sent: Thursday, January 14, 2016 1:06 PM
>>>>> To: MarkLogic Developer Discussion
>>>>> Subject: [MarkLogic Dev General] CPF pre-commit action?
>>>>> 
>>>>> Is it possible to configure a CPF pipeline such that when a 
>>>>>document is  inserted, transformations are first executed on the 
>>>>>document in a  pre-commit stage, and if those complete 
>>>>>successfully, then the  transformation result is what's finally 
>>>>>committed at that URI? And any  remaining pipeline could carry on 
>>>>>without the pre-transformed data ever  being visible to the 
>>>>>database?
>>>>> 
>>>>> -Will
>>>>> _______________________________________________
>>>>> General mailing list
>>>>> General at developer.marklogic.com
>>>>> Manage your subscription at:
>>>>> http://developer.marklogic.com/mailman/listinfo/general
>>>>> _______________________________________________
>>>>> General mailing list
>>>>> General at developer.marklogic.com
>>>>> Manage your subscription at:
>>>>> http://developer.marklogic.com/mailman/listinfo/general
>>>> 
>>>> _______________________________________________
>>>> General mailing list
>>>> General at developer.marklogic.com
>>>> Manage your subscription at:
>>>> http://developer.marklogic.com/mailman/listinfo/general
>>> 
>>> _______________________________________________
>>> General mailing list
>>> General at developer.marklogic.com
>>> Manage your subscription at:
>>> http://developer.marklogic.com/mailman/listinfo/general
>> 
>> _______________________________________________
>> General mailing list
>> General at developer.marklogic.com
>> Manage your subscription at:
>> http://developer.marklogic.com/mailman/listinfo/general
>
>_______________________________________________
>General mailing list
>General at developer.marklogic.com
>Manage your subscription at:
>http://developer.marklogic.com/mailman/listinfo/general

_______________________________________________
General mailing list
General at developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general


More information about the General mailing list