[MarkLogic Dev General] xqsync throughput

Michael Blakeley mike at blakeley.com
Wed Mar 14 07:14:37 PDT 2012


I would expect better than that. What is the document rate?

You may not have enough client threads to keep the servers busy. What does the utilization look like on both sides?

You may also be memory-limited in the JVM at some point, especially if the documents are big. If so, the JVM will spend a lot of time running the garbage collector. You can check that idea with the '-verbose:gc' option.

Could there be a network limitation other than bandwidth? You might check that by exporting to packages instead, and see what that performance looks like. I have seen some cases where there was a slow hop on the network, or where a firewall was limiting performance.

-- Mike

On 14 Mar 2012, at 13:38 , Mike Sokolov wrote:

> I wonder if anyone has a rough guide to what sort of transfer speeds can 
> be expected using xqsync to transfer a database from one node to 
> another.  I have two quite beefy servers on the same LAN (at least 
> 100Mb/s ~ 12MB/s), and I'm only getting ~30kB/sec.  I was hoping to get 
> a few orders of magnitude more, but am I smoking crack?  Is there 
> something I could be doing or not doing that might be limiting the speed 
> somehow?
> 
> This is my setup:
> 
> java -cp ${BIN}/xqsync.jar:$BIN/xcc.jar:$BIN/xstream.jar:$BIN/xpp3.jar 
> -Xmx1024m \
>  -DINPUT_CONNECTION_STRING=$SRCDB \
>  -DOUTPUT_CONNECTION_STRING=$DSTDB \
>  -DSKIP_EXISTING=true \
>  -DCOPY_COLLECTIONS=false \
>  -DCOPY_PERMISSIONS=false \
>  -DCOPY_PROPERTIES=true \
>  -DCOPY_QUALITY=false \
>  -DINPUT_BATCH_SIZE=10 \
>  -DINPUT_QUERY_CACHABLE \
>  -DTHREADS=8  \
>     com.marklogic.ps.xqsync.XQSync
> 
> These are the startup messages from the log:
> 
> INFO: XQSync starting: version 2009-03-10.1 on 1.6.0_26 (Java(TM) SE 
> Runtime Environment)
> Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSync main
> INFO: XCC version = 3.2-7
> Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager run
> INFO: starting pool of 8 threads, queue size = 10000
> Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.Monitor run
> INFO: starting
> Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager run
> INFO: output version info: client 3.2-7, server 4.1-11
> Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager run
> INFO: input version info: client 3.2-7, server 4.1-11
> Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager 
> queueFromInputConnection
> INFO: buffer size = 0, caching = false
> Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager getUrisRequest
> INFO: listing all documents (with uri lexicon)
> Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager 
> queueFromInputConnection
> 
> The connector is a bit old: Can I expect any substantial improvement 
> from updating that?
> 
> -- 
> Michael Sokolov
> Engineering Director
> www.ifactory.com
> 
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> http://developer.marklogic.com/mailman/listinfo/general
> 



More information about the General mailing list