<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html; charset=ISO-8859-1"
 http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
Nick - you can also create range indexes explicitly in MarkLogic, and
these will really help with the performance of joins, just as they do
in eXist.<br>
<br>
-Mike<br>
<br>
On 03/16/2012 08:26 AM, Nick Tuckett wrote:
<blockquote
 cite="mid:CALGwMTPBBu2HrprdadRb4CSkLtauzAw3_N2_K_eOiJ7EeD9DYA@mail.gmail.com"
 type="cite">I'm evaluating MarkLogic as a possible way to store and
access around 25Mb (and growing) of fairly complex XML data. For one
particular type of common query for my application, I'm seeing
drastically different performance between MarkLogic and eXist.&nbsp;
I would be very grateful for any feedback or advice on how to improve
this performance
  <div><br>
  </div>
  <div>One common feature of this data are attributes containing
identifying values that reference other elements in the collection - an
example of this is for referencing localised text from a common XML
file.&nbsp;I have been using a fairly simple query to benchmark performance
that looks like this:
  <div><br>
  </div>
  <div>
  <div>for $x in
collection('/db/content')//(elementa|elementb|elementc|elementd|elemente|elementf)</div>
  <div>let $e :=
doc('/db/language/lang_en.xml')//text[@id=$x/@localisedtextid]</div>
  <div>order by $x/@id</div>
  <div>return&nbsp;<br>
  </div>
  <div>&lt;localisedtext quest="{$e}"/&gt;&nbsp;</div>
  <div><br>
  </div>
  </div>
  <div>The benchmark content has around 2500 instances for this
particular case. With everything else constant (hardware, OS, content)
I see drastically different performance between MarkLogic and eXist.
The former takes around 59 seconds to return the data for all
instances, the latter takes 8 seconds.</div>
  <div><br>
  </div>
  <div>As I understand it, MarkLogic sets up indexing automatically,
including indexing on element-attribute pairs. To match this, I created
an explicit equivalent index for eXist for the text/@id pair for use in
this case.</div>
  <div><br>
  </div>
  <div>For MarkLogic, running the query with the profiler showed that
around 75% of the execution time went on '@id = $x/@localisedtextid',
and query tracing produced the following output:</div>
  <div><br>
  </div>
  <div>Initial part of query:</div>
  <div><br>
  </div>
  <div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10:
xdmp:eval("xdmp:query-trace(true()),&amp;#10;for $x in
collection('/db/content...", (), &lt;options
xmlns="xdmp:eval"&gt;&lt;database&gt;1488253557778688591&lt;/database&gt;&lt;modules&gt;148825355777868...&lt;/options&gt;)</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10: Analyzing
path for $x:
fn:collection("/db/content")/descendant-or-self::node()/(elementa|elementb|elementc|elementd|elemente|elementf)</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10: Step 1 is
searchable: fn:collection("/db/content")</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10: Step 2 does
not use indexes: descendant-or-self::node()</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10: Step 3 is
searchable:&nbsp;(elementa|elementb|elementc|elementd|elemente|elementf)</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10: Path is
fully searchable.</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10: Gathering
constraints.</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10: Step 1
contributed 1 constraint: fn:collection("/db/
content")</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:48: Step 3
contributed 1 constraint:&nbsp;elementa</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:62: Step 3
contributed 1 constraint:&nbsp;elementb</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:81: Step 3
contributed 1 constraint:&nbsp;elementc</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:110: Step 3
contributed 1 constraint: elementd</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:122: Step 3
contributed 1 constraint: elemente</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:132: Step 3
contributed 1 constraint: elementf</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10: Executing
search.<br>
  </div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 2:10: Selected 8
fragments to filter.</div>
  </div>
  <div><br>
  </div>
  <div>Iterated part of query (repeat N times...)</div>
  <div><br>
  </div>
  <div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48:
xdmp:eval("xdmp:query-trace(true()),&amp;#10;for $x in
collection('/db/content...", (), &lt;options
xmlns="xdmp:eval"&gt;&lt;database&gt;1488253557778688591&lt;/database&gt;&lt;modules&gt;148825355777868...&lt;/options&gt;)</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Analyzing
path: fn:doc("/db/language/lang_en.xml")/descendant::text[@id =
xs:untypedAtomic("ElementName143001")]</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Step 1 is
searchable: fn:doc("/db/language/lang_en.xml")</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Step 2 is
searchable: descendant::text[@id =
xs:untypedAtomic("ElementName143001")]</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Path is
fully searchable.</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Gathering
constraints.</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:10: Step 1
contributed 1 constraint: fn:doc("/db/language/lang_en.xml")</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:54: Comparison
contributed hash value constraint: text/@id =
xs:untypedAtomic("ElementName143001")</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Step 2
predicate 1 contributed 1 constraint: @id =
xs:untypedAtomic("ElementName143001")</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:54: Comparison
contributed hash value constraint: text/@id =
xs:untypedAtomic("ElementName143001")</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Step 2
predicate 1 contributed 1 constraint: @id =
xs:untypedAtomic("ElementName143001")</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Step 2
contributed 2 constraints: descendant::text[@id =
xs:untypedAtomic("ElementName143001")]</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Executing
search.</div>
  <div>2012-03-16 11:35:35.743 Info: App-Services: at 3:48: Selected 1
fragment to filter</div>
  </div>
  <div><br>
  </div>
  <div>Query meters:</div>
  <div><br>
  </div>
  <div>
  <div>&nbsp; &lt;qm:query-meters xsi:schemaLocation="<a
 moz-do-not-send="true" href="http://marklogic.com/xdmp/query-meters">http://marklogic.com/xdmp/query-meters</a>
query-meters.xsd" xmlns:qm="<a moz-do-not-send="true"
 href="http://marklogic.com/xdmp/query-meters">http://marklogic.com/xdmp/query-meters</a>"
xmlns:xsi="<a moz-do-not-send="true"
 href="http://www.w3.org/2001/XMLSchema-instance">http://www.w3.org/2001/XMLSchema-instance</a>"&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:elapsed-time&gt;PT56.655659S&lt;/qm:elapsed-time&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:requests&gt;0&lt;/qm:requests&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:list-cache-hits&gt;1043&lt;/qm:list-cache-hits&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:list-cache-misses&gt;0&lt;/qm:list-cache-misses&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:in-memory-list-hits&gt;0&lt;/qm:in-memory-list-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:expanded-tree-cache-hits&gt;519&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:compressed-tree-cache-hits&gt;0&lt;/qm:compressed-tree-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:compressed-tree-cache-misses&gt;0&lt;/qm:compressed-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:in-memory-compressed-tree-hits&gt;0&lt;/qm:in-memory-compressed-tree-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:value-cache-hits&gt;6672643&lt;/qm:value-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:value-cache-misses&gt;6673683&lt;/qm:value-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:regexp-cache-hits&gt;0&lt;/qm:regexp-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:regexp-cache-misses&gt;0&lt;/qm:regexp-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:link-cache-hits&gt;0&lt;/qm:link-cache-hits&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:link-cache-misses&gt;0&lt;/qm:link-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:filter-hits&gt;0&lt;/qm:filter-hits&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:filter-misses&gt;0&lt;/qm:filter-misses&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:fragments-added&gt;0&lt;/qm:fragments-added&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:fragments-deleted&gt;0&lt;/qm:fragments-deleted&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:fs-program-cache-hits&gt;0&lt;/qm:fs-program-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:fs-program-cache-misses&gt;1&lt;/qm:fs-program-cache-misses&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:db-program-cache-hits&gt;0&lt;/qm:db-program-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:db-program-cache-misses&gt;0&lt;/qm:db-program-cache-misses&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:env-program-cache-hits&gt;0&lt;/qm:env-program-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:env-program-cache-misses&gt;0&lt;/qm:env-program-cache-misses&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:fs-main-module-sequence-cache-hits&gt;0&lt;/qm:fs-main-module-sequence-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:fs-main-module-sequence-cache-misses&gt;0&lt;/qm:fs-main-module-sequence-cache-misses&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:db-main-module-sequence-cache-hits&gt;0&lt;/qm:db-main-module-sequence-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:db-main-module-sequence-cache-misses&gt;0&lt;/qm:db-main-module-sequence-cache-misses&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:fs-library-module-cache-hits&gt;0&lt;/qm:fs-library-module-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:fs-library-module-cache-misses&gt;0&lt;/qm:fs-library-module-cache-misses&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:db-library-module-cache-hits&gt;0&lt;/qm:db-library-module-cache-hits&gt;</div>
  <div>&nbsp; &nbsp;
&lt;qm:db-library-module-cache-misses&gt;0&lt;/qm:db-library-module-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:fragments&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:fragment&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:root&gt;contents&lt;/qm:root&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;511&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:fragment&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:fragment&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:root&gt;database&lt;/qm:root&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;8&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:fragment&gt;</div>
  <div>&nbsp; &nbsp; &lt;/qm:fragments&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:documents&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:document&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:uri&gt;/db/content/file1.xml&lt;/qm:uri&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;1&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:document&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:document&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:uri&gt;/db/content/file2.xml&lt;/qm:uri&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;1&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:document&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:document&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:uri&gt;/db/language/lang_en.xml&lt;/qm:uri&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;511&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:document&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:document&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:uri&gt;/db/content/file3.xml&lt;/qm:uri&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;1&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:document&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:document&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:uri&gt;/db/content/file4.xml&lt;/qm:uri&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;1&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:document&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:document&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:uri&gt;/db/content/file5.xml&lt;/qm:uri&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;1&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:document&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:document&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:uri&gt;/db/content/file6.xml&lt;/qm:uri&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;1&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:document&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:document&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:uri&gt;/db/content/file7.xml&lt;/qm:uri&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;1&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:document&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;qm:document&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:uri&gt;/db/content/file8.xml&lt;/qm:uri&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-hits&gt;1&lt;/qm:expanded-tree-cache-hits&gt;</div>
  <div><span class="Apple-tab-span" style="white-space: pre;"> </span>&lt;qm:expanded-tree-cache-misses&gt;0&lt;/qm:expanded-tree-cache-misses&gt;</div>
  <div>&nbsp; &nbsp; &nbsp; &lt;/qm:document&gt;</div>
  <div>&nbsp; &nbsp; &lt;/qm:documents&gt;</div>
  <div>&nbsp; &nbsp; &lt;qm:hosts/&gt;</div>
  <div>&nbsp; &lt;/qm:query-meters&gt;</div>
  </div>
  </div>
  <pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
General mailing list
<a class="moz-txt-link-abbreviated" href="mailto:General@developer.marklogic.com">General@developer.marklogic.com</a>
<a class="moz-txt-link-freetext" href="http://developer.marklogic.com/mailman/listinfo/general">http://developer.marklogic.com/mailman/listinfo/general</a>
  </pre>
</blockquote>
</body>
</html>