[MarkLogic Dev General] (no subject)

Justin Makeig Justin.Makeig at marklogic.com
Wed Jan 24 12:15:39 PST 2018


You’re trying to update all of the documents in the “input” collection in a single transaction, the default scope of a JavaScript module. For small numbers of documents (hundreds or thousands) that will work. For large or unknown numbers of documents that will generally overwhelm some resource. In your case, you’ve blown out your expanded tree cache. Your best bet is to break the transformation into multiple transactions and spread those out over multiple hosts in a cluster. The MarkLogic Data Movement SDK is designed for exactly such workloads. Take a look at the docs <https://docs.marklogic.com/guide/java/data-movement#id_51555> for more information on how to orchestrate a bulk transformation from Java.

Justin

--
Justin Makeig
Senior Director, Product Management
MarkLogic
jmakeig at marklogic.com<mailto:jmakeig at marklogic.com>

On Jan 24, 2018, at 10:19 AM, C. Yaswanth <rockingyas at gmail.com<mailto:rockingyas at gmail.com>> wrote:

Hi All,

Actually i have a set of json files(i.e.Total : 1M with Size 500MB). Each json file has 18 Keys. I tried to implement Envelope pattern using below Javascript

    'use strict';
    declareUpdate()
    var docs = fn.collection("input");
    for(var doc of docs) {
      var transformed = {};

      transformed.Metadata = { "Last Used" : ""};
      transformed.Updated = { "University" : "UCLA"}
      transformed.Source = doc; //Sending original data under Source section
      xdmp.nodeReplace(doc,transformed)
    }

I tried invoking this `JS.sjs` using JAVA API of marklogic 9. But i encountered below error :

    Exception in thread "main" com.marklogic.client.FailedRequestException: Local message: failed to apply resource at invoke: Internal Server Error. Server Message: XDMP-EXPNTREECACHEFULL: for(var doc of docs) { -- Expanded tree cache full on host localhost uri file.json-0-968991
     at com.marklogic.client.impl.OkHttpServices.checkStatus(OkHttpServices.java:4317)
     at com.marklogic.client.impl.OkHttpServices.postIteratedResourceImpl(OkHttpServices.java:3831)
     at com.marklogic.client.impl.OkHttpServices.postEvalInvoke(OkHttpServices.java:3768)
     at com.marklogic.client.impl.ServerEvaluationCallImpl.eval(ServerEvaluationCallImpl.java:164)
     at com.marklogic.client.impl.ServerEvaluationCallImpl.eval(ServerEvaluationCallImpl.java:153)
     at com.marklogic.client.impl.ServerEvaluationCallImpl.evalAs(ServerEvaluationCallImpl.java:144)
     at bulkimport.Tsm.main(Tsm.java:19)

I went through documentation (i.e.https://help.marklogic.com/knowledgebase/article/View/9/16/resolving-xdmp-expntreecachefull-errors)  where they had mentioned ways to resolve this error. Following that i had increased `expanded tree cache size*` to `2048` but still i am facing same error.

How can i optimize by above code (i.e.`JS.sjs`) to avoid this error ?

Any help is appreciated.
_______________________________________________
General mailing list
General at developer.marklogic.com<mailto:General at developer.marklogic.com>
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://developer.marklogic.com/pipermail/general/attachments/20180124/826d9abf/attachment.html 


More information about the General mailing list