[MarkLogic Dev General] xqsync throughput

Mike Sokolov sokolov at ifactory.com
Wed Mar 14 06:38:45 PDT 2012


I wonder if anyone has a rough guide to what sort of transfer speeds can 
be expected using xqsync to transfer a database from one node to 
another.  I have two quite beefy servers on the same LAN (at least 
100Mb/s ~ 12MB/s), and I'm only getting ~30kB/sec.  I was hoping to get 
a few orders of magnitude more, but am I smoking crack?  Is there 
something I could be doing or not doing that might be limiting the speed 
somehow?

This is my setup:

java -cp ${BIN}/xqsync.jar:$BIN/xcc.jar:$BIN/xstream.jar:$BIN/xpp3.jar 
-Xmx1024m \
  -DINPUT_CONNECTION_STRING=$SRCDB \
  -DOUTPUT_CONNECTION_STRING=$DSTDB \
  -DSKIP_EXISTING=true \
  -DCOPY_COLLECTIONS=false \
  -DCOPY_PERMISSIONS=false \
  -DCOPY_PROPERTIES=true \
  -DCOPY_QUALITY=false \
  -DINPUT_BATCH_SIZE=10 \
  -DINPUT_QUERY_CACHABLE \
  -DTHREADS=8  \
     com.marklogic.ps.xqsync.XQSync

These are the startup messages from the log:

INFO: XQSync starting: version 2009-03-10.1 on 1.6.0_26 (Java(TM) SE 
Runtime Environment)
Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSync main
INFO: XCC version = 3.2-7
Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager run
INFO: starting pool of 8 threads, queue size = 10000
Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.Monitor run
INFO: starting
Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager run
INFO: output version info: client 3.2-7, server 4.1-11
Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager run
INFO: input version info: client 3.2-7, server 4.1-11
Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager 
queueFromInputConnection
INFO: buffer size = 0, caching = false
Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager getUrisRequest
INFO: listing all documents (with uri lexicon)
Mar 14, 2012 2:33:36 PM com.marklogic.ps.xqsync.XQSyncManager 
queueFromInputConnection

The connector is a bit old: Can I expect any substantial improvement 
from updating that?

-- 
Michael Sokolov
Engineering Director
www.ifactory.com



More information about the General mailing list