[MarkLogic Dev General] CPF pre-commit action?

Will Thompson wthompson at oconnors.com
Mon Jan 18 09:33:37 PST 2016


Geert,

I hadn't considered that, but it would definitely achieve the desired behavior. Thanks for the suggestion. It would add a lot more complexity to the pipeline, but its sounds feasible.

-Will

> On Jan 15, 2016, at 1:38 PM, Geert Josten <Geert.Josten at marklogic.com> wrote:
> 
> Have you considered taking an approach similar to Info Studio: insert
> unprocessed docs in an separate database in which you have all the CPF
> pipelines you need. Not until finishing of processing it gets copied into
> the target database. Info Studio was using the Fab database for that
> purpose..
> 
> Cheers
> 
> On 1/15/16, 5:06 PM, "general-bounces at developer.marklogic.com on behalf of
> Will Thompson" <general-bounces at developer.marklogic.com on behalf of
> wthompson at oconnors.com> wrote:
> 
>> Geert,
>> 
>> We're using CPF because some steps may necessitate human intervention, in
>> which case they go into a queue, get resolved, a state change occurs, and
>> the pipeline marches on. But are you suggesting a pre-commit trigger to
>> supplement the existing CPF pipeline?
>> 
>> Danny,
>> 
>> The main issue for us is user experience. For example, in most cases
>> document X will already exist in the database and users can see it as
>> part of the application. We want to replace document X with an updated
>> version and not have it appear visible to users until it has been
>> transformed. Hiding it in a collection would hide it from users as well.
>> I said "version" so maybe DLS seems like a good fit? But I'm still unsure
>> of how to begin the pipeline with the document in a checked-out state.
>> 
>> We do have a solution to this, which is essentially: the application
>> takes care of it. But I wanted to see if it were possible to push that
>> logic back into CPF and simplify our API such that other applications
>> could utilize the pipeline with basic insert and delete operations.
>> 
>> Thanks,
>> 
>> Will
>> 
>> 
>>> On Jan 15, 2016, at 5:26 AM, Geert Josten <Geert.Josten at marklogic.com>
>>> wrote:
>>> 
>>> Or how about KISS: just use one pre-commit trigger that does all the
>>> transformation/processing in one blow..
>>> 
>>> Kind regards,
>>> Geert
>>> 
>>> On 1/15/16, 12:42 AM, "general-bounces at developer.marklogic.com on behalf
>>> of Danny Sokolsky" <general-bounces at developer.marklogic.com on behalf of
>>> Danny.Sokolsky at marklogic.com> wrote:
>>> 
>>>> Hi Will,
>>>> 
>>>> I can't think of a way to do this without modifying the core cpf code.
>>>> For some of the status-change-handling pipeline steps, there are
>>>> pre-commit triggers, but those are for cpf bookkeeping mostly, I
>>>> believe.
>>>> And I would not recommend changing that code.
>>>> 
>>>> But I would question why you would want to do this.  CPF is designed to
>>>> be resilient and to allow you to have multiple steps in your pipeline.
>>>> Why not make a step that does the transform its own step?  If you do
>>>> not
>>>> want the document to be visible until it is transformed, then put it in
>>>> some collection initially that is not seen by your application, and
>>>> then
>>>> take it out of that collection when you are done with the transform (or
>>>> something similar to that).
>>>> 
>>>> Is it just because you want to have one less step in your pipeline that
>>>> you want to do this?
>>>> 
>>>> -Danny
>>>> 
>>>> -----Original Message-----
>>>> From: general-bounces at developer.marklogic.com
>>>> [mailto:general-bounces at developer.marklogic.com] On Behalf Of Will
>>>> Thompson
>>>> Sent: Thursday, January 14, 2016 1:06 PM
>>>> To: MarkLogic Developer Discussion
>>>> Subject: [MarkLogic Dev General] CPF pre-commit action?
>>>> 
>>>> Is it possible to configure a CPF pipeline such that when a document is
>>>> inserted, transformations are first executed on the document in a
>>>> pre-commit stage, and if those complete successfully, then the
>>>> transformation result is what's finally committed at that URI? And any
>>>> remaining pipeline could carry on without the pre-transformed data ever
>>>> being visible to the database?
>>>> 
>>>> -Will
>>>> _______________________________________________
>>>> General mailing list
>>>> General at developer.marklogic.com
>>>> Manage your subscription at:
>>>> http://developer.marklogic.com/mailman/listinfo/general
>>>> _______________________________________________
>>>> General mailing list
>>>> General at developer.marklogic.com
>>>> Manage your subscription at:
>>>> http://developer.marklogic.com/mailman/listinfo/general
>>> 
>>> _______________________________________________
>>> General mailing list
>>> General at developer.marklogic.com
>>> Manage your subscription at:
>>> http://developer.marklogic.com/mailman/listinfo/general
>> 
>> _______________________________________________
>> General mailing list
>> General at developer.marklogic.com
>> Manage your subscription at:
>> http://developer.marklogic.com/mailman/listinfo/general
> 
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at: 
> http://developer.marklogic.com/mailman/listinfo/general



More information about the General mailing list