[MarkLogic Dev General] Java XCC save query result to file

Ron Hitchens ron at ronsoft.com
Tue Mar 20 14:32:30 PDT 2012

   While Mike is right (he always is, basically), the XQuery-side
streaming he's describing is slightly different than XCC's definition
of streaming.

   By default, XCC issues your request to MarkLogic then reads the
entire result into memory before returning from the submitRequest()
method.  By the time it returns, it has released the connection and
MarkLogic is done with you.  The entire result is cached in Java

   This is XCC's "cache mode" - not streaming - and it can blow
memory if your XQuery returns too much data.  But because it's
all buffered, it means you can access ResultItems randomly and
repeatedly in the ResultSequence.  Cached mode is more efficient
and is appropriate for the common use case of fetching results to
service a web request.

   You can set an option on your XCC request to specify that 
you want "non-cached mode" - streaming.  In that case, XCC does
not automatically read the full response.  It waits for you to
fetch the items from the ResultSequence and reads the response
from MarkLogic as needed.  If you read some/write some, then
you can pass large amounts of data through without holding it
all locally.  Even better is to use the writeTo() method which
will do that for you.

   But if you're streaming and don't fetch the result items in
a timely manner, MarkLogic may hang up the connection on you even
if the query result is not especially large.

   It's also important to close streaming ResultSequences after
use, since they are holding the underlying connection which can't
be re-used until you do.

   I guess I still remember all that stuff.

On Mar 20, 2012, at 9:01 PM, Michael Blakeley wrote:

> Whether or not a result is capable of streaming will depend on the XQuery too. I just finishing writing something about this:
>    http://blakeley.com/blogofile/2012/03/19/let-free-style-and-streaming/
> -- Mike
> On 20 Mar 2012, at 13:58 , David Lee wrote:
>> How critical to your app is it that the result is "streamed" ?
>> Is it always XML or is it sometimes text or binary ?
>> The xmlsh MarkLogic extension "get" command can be used as an example of writing the result items to a Java stream.
>> The "writeResult" method implements the guts of this command
>> http://xmlsh.svn.sourceforge.net/viewvc/xmlsh/extensions/marklogic/src/org/xmlsh/marklogic/util/MLCommand.java?revision=671&view=markup
>> The key is that a ResultSequence is an iterator of XdmItem.
>> Each XdmItem has a "writeTo" method.
>> There is no guarentee that this is "streamed" to my knowledge but it may well be.
>> -----------------------------------------------------------------------------
>> David Lee
>> Lead Engineer
>> MarkLogic Corporation
>> dlee at marklogic.com
>> Phone: +1 650-287-2531
>> Cell:  +1 812-630-7622
>> www.marklogic.com
>> This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation.
>> From: general-bounces at developer.marklogic.com [mailto:general-bounces at developer.marklogic.com] On Behalf Of Gary Larsen
>> Sent: Tuesday, March 20, 2012 4:38 PM
>> To: 'General MarkLogic Developer Discussion'
>> Subject: [MarkLogic Dev General] Java XCC save query result to file
>> Hi,
>> For some queries I’m using the xdmp:save() function to store results to file.   This will not always work because the MarkLogic server may not have access to the share location.
>> Instead the results will need to be saved by the application by streaming the ResultSequence returned from session.submitRequest(request) call.
>> Is there a preferred way to do this?
>> Thanks for your advice.  An example would be helpful if you have one.
>> Gary
>> _______________________________________________
>> General mailing list
>> General at developer.marklogic.com
>> http://developer.marklogic.com/mailman/listinfo/general
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> http://developer.marklogic.com/mailman/listinfo/general

Ron Hitchens {mailto:ron at ronsoft.com}   Ronsoft Technologies
     +44 7879 358 212 (voice)          http://www.ronsoft.com
     +1 707 924 3878 (fax)              Bit Twiddling At Its Finest
"No amount of belief establishes any fact." -Unknown

More information about the General mailing list