[MarkLogic Dev General] tail-recursion with xdmp:spawn
wfeick at marklogic.com
Wed Apr 7 15:45:21 PDT 2010
That can works too, but you should be sure to consider whether documents
[N,N+1000) at the time of the spawn are guaranteed to still be [N,N
+1000] when the task gets around to executing. If you're adding,
removing, or modifying documents, you'll get different document ordering
On Wed, 2010-04-07 at 07:53 -0700, Geert Josten wrote:
> > Another approach is to have an initial query create a list of
> > documents to process and cut it into chunks (say 1000
> > documents each) that are each handed off to a spawned task.
> > With this, the configured number of threads in the task queue
> > will run in parallel giving you higher overall throughput.
> That is what I more or less did. However, I decided to pass in a start
> and end count, to have the function find out for itself which
> documents that should be.
> And one other thing I did was divide the total range into a number of
> segments equal to the number of parallel threads I would want to run,
> which like this can be smaller than the total number of threads
> allowed on the task server, giving more room for normal traffic. So
> instead of spawning all tasks at once (cluttering the queue as well),
> each task creates the next one. A simple try catch makes sure the
> recursive spawning isn't terminated before the end..
> Kind regards,
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the General