[MarkLogic Dev General] (no subject)

C. Yaswanth rockingyas at gmail.com
Thu Jan 25 19:34:12 PST 2018


Thanks Justin,

As, You suggested i went through MarkLogic Data Movement SDK  in java api
but i find two ways of invoking a transform module .

           1) One way is to install a transform in the server and then
using Data movement SDK.(i.e.http://docs.marklogic.com/guide/java/transforms
)
            2)Other way is to install invoke a module .(i.e.
https://docs.marklogic.com/guide/java/resourceservices#id_84134)

I want to know , using first way only i can implement SDK or even if use
other way too i can do that .

Thanks
Yaswanth C

On Thu, Jan 25, 2018 at 1:45 AM, Justin Makeig <Justin.Makeig at marklogic.com>
wrote:

> You’re trying to update all of the documents in the “input” collection in
> a single transaction, the default scope of a JavaScript module. For small
> numbers of documents (hundreds or thousands) that will work. For large or
> unknown numbers of documents that will generally overwhelm some resource.
> In your case, you’ve blown out your expanded tree cache. Your best bet is
> to break the transformation into multiple transactions and spread those out
> over multiple hosts in a cluster. The MarkLogic Data Movement SDK is
> designed for exactly such workloads. Take a look at the docs <
> https://docs.marklogic.com/guide/java/data-movement#id_51555> for more
> information on how to orchestrate a bulk transformation from Java.
>
> Justin
>
> --
> Justin Makeig
> Senior Director, Product Management
> MarkLogic
> jmakeig at marklogic.com
>
> On Jan 24, 2018, at 10:19 AM, C. Yaswanth <rockingyas at gmail.com> wrote:
>
> Hi All,
>
> Actually i have a set of json files(i.e.Total : 1M with Size 500MB). Each
> json file has 18 Keys. I tried to implement Envelope pattern using below
> Javascript
>
>     'use strict';
>     declareUpdate()
>     var docs = fn.collection("input");
>     for(var doc of docs) {
>       var transformed = {};
>
>       transformed.Metadata = { "Last Used" : ""};
>       transformed.Updated = { "University" : "UCLA"}
>       transformed.Source = doc; //Sending original data under Source
> section
>       xdmp.nodeReplace(doc,transformed)
>     }
>
> I tried invoking this `JS.sjs` using JAVA API of marklogic 9. But i
> encountered below error :
>
>     Exception in thread "main" com.marklogic.client.FailedRequestException:
> Local message: failed to apply resource at invoke: Internal Server Error.
> Server Message: XDMP-EXPNTREECACHEFULL: for(var doc of docs) { -- Expanded
> tree cache full on host localhost uri file.json-0-968991
>      at com.marklogic.client.impl.OkHttpServices.checkStatus(OkHttpS
> ervices.java:4317)
>      at com.marklogic.client.impl.OkHttpServices.postIteratedResourc
> eImpl(OkHttpServices.java:3831)
>      at com.marklogic.client.impl.OkHttpServices.postEvalInvoke(OkHt
> tpServices.java:3768)
>      at com.marklogic.client.impl.ServerEvaluationCallImpl.eval(Serv
> erEvaluationCallImpl.java:164)
>      at com.marklogic.client.impl.ServerEvaluationCallImpl.eval(Serv
> erEvaluationCallImpl.java:153)
>      at com.marklogic.client.impl.ServerEvaluationCallImpl.evalAs(Se
> rverEvaluationCallImpl.java:144)
>      at bulkimport.Tsm.main(Tsm.java:19)
>
> I went through documentation (i.e.https://help.marklogic.
> com/knowledgebase/article/View/9/16/resolving-xdmp-expnt
> reecachefull-errors)  where they had mentioned ways to resolve this
> error. Following that i had increased `expanded tree cache size*` to `2048`
> but still i am facing same error.
>
> How can i optimize by above code (i.e.`JS.sjs`) to avoid this error ?
>
> Any help is appreciated.
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
>
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at:
> http://developer.marklogic.com/mailman/listinfo/general
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://developer.marklogic.com/pipermail/general/attachments/20180126/9cc6c885/attachment.html 


More information about the General mailing list