[MarkLogic Dev General] Struggling with Query Time Out

Jonathan Cook jonathan.cook2 at bbc.co.uk
Tue Mar 20 02:58:05 PDT 2012


You could have a look at this as well:
http://joncook.github.com/blog/2012/02/12/evaluating-mark-logic-xquery-perfo
rmance/

Which will help you figure out which bit is taking the longest.

Jon


On 19/03/2012 20:16, "Michael Blakeley" <mike at blakeley.com> wrote:

> Betty, I think it's the '[1]' that makes that expression unsearchable.
> Normally the XPath indexes simply record the presence of elements, not their
> position.
> 
> -- Mike
> 
> On 16 Mar 2012, at 15:03 , Betty Harvey wrote:
> 
>> Thanks!!!
>> 
>> I set an element range index on the main database and have apparently run
>> out of disk space - I will deal with that issue later. It is running on a
>> VM machine.
>> 
>> I also set a range index on EventDate in the 'documents' database for test
>> purposes.   I rewrote the query to use cts:search and it comes back on the
>> 'documents' database that the "Expression is unsearchable" so it looks
>> like
>> I am not sure what this error message means but I think it might not be
>> recognizing the range index.
>> 
>> Am I missing something significant.   The documents have 3 namespaces.
>> The EventDate is in the 'ns1' namespace.  I only used one
>> cts:element-range-query as a test.
>> 
>> Revised test code:
>> 
>> for $ACE in
>> cts:search(collection()/descendant::ns1:ACE/ns1:EventSet/ns1:GeneralEvent[1],
>>     cts:element-range-query (xs:QName('EventDate'), '<',
>> xs:dateTime('2011-03-01T00:00:00') ) )
>> let $ACEId := $ACE/ancestor::ns1:ACE/ns1:ACEId
>> let $EventDate := $ACE/ns1:EventDate
>> return
>> <a>
>> {$ACEId}
>> {$EventDate}
>> <time>{xdmp:elapsed-time()}</time>
>> </a>
>> 
>>> Hi Betty,
>>> 
>>> Using a cts:search like David suggests could speed up considerably,
>>> indeed. You can use xdmp:directory as searchable expression, I thought,
>>> but you can also add it to the query part using cts:directory-query.
>>> 
>>> Note though that if you rewrite the date predicates to
>>> cts:element-range-query's, that it may make a lot of difference whether
>>> ACE is a fragment root or not. If you include /descendant::ACE in your
>>> searchable path, then the end result is filtered to make sure each ACE
>>> matches the query, but there could be a lot of false positives (and hence
>>> xdmp:estimate could return a too high value).
>>> 
>>> Kind regards,
>>> Geert
>>> 
>>>> -----Oorspronkelijk bericht-----
>>>> Van: general-bounces at developer.marklogic.com [mailto:general-
>>>> bounces at developer.marklogic.com] Namens David Lee
>>>> Verzonden: vrijdag 16 maart 2012 19:54
>>>> Aan: MarkLogic Developer Discussion
>>>> Onderwerp: Re: [MarkLogic Dev General] Struggling with Query Time Out
>>>> 
>>>> First off cts:search is exactly what you want for this.
>>>> Second you are doing string compares against datetime values.  To help
>>> with this
>>>> you may need to create a range index  on EventDate and compare against
>>>> xs:dateTime('xxxxxx')
>>>> Thirdly your doing a directory search which you might not actually need
>>> if these
>>>> documents are in know namespaces.
>>>> But hold off on that until you get the first two worked out.
>>>> cts:search() is really your friend in this case, but you do want to make
>>> a range
>>>> index so that the system knows the values are dates otherwise "gt" will
>>> do string
>>>> not date comparisons
>>>> 
>>>> Once you get both those working your searches should be nearly instant.
>>>> 
>>>> 
>>>> 
>>> --------------------------------------------------------------------------
>>> ---
>>>> David Lee
>>>> Lead Engineer
>>>> MarkLogic Corporation
>>>> dlee at marklogic.com
>>>> Phone: +1 650-287-2531
>>>> Cell:  +1 812-630-7622
>>>> www.marklogic.com
>>>> 
>>>> This e-mail and any accompanying attachments are confidential. The
>>>> information is intended solely for the use of the individual to whom it
>>> is
>>>> addressed. Any review, disclosure, copying, distribution, or use of this
>>> e-mail
>>>> communication by others is strictly prohibited. If you are not the
>>> intended
>>>> recipient, please notify us immediately by returning this message to the
>>> sender
>>>> and delete all copies. Thank you for your cooperation.
>>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: general-bounces at developer.marklogic.com [mailto:general-
>>>>> bounces at developer.marklogic.com] On Behalf Of Betty Harvey
>>>>> Sent: Friday, March 16, 2012 3:17 PM
>>>>> To: MarkLogic Developer Discussion
>>>>> Subject: [MarkLogic Dev General] Struggling with Query Time Out
>>>>> 
>>>>> I have been unable to get this query to run successfully without
>>> timing
>>>>> out.  To make sure my logic was correct I placed 100 documents in the
>>>>> 'documents' database and query runs successfully and very quickly. In
>>> the
>>>>> large database 1.7 million objects the query always times out.
>>>>> 
>>>>> I am not sure cts:search will help.  I played around with it without
>>>>> success.   The goal of the query is to gather information for a
>>> particular
>>>>> month based on when the document was created.   Below is the code:
>>>>> 
>>>>> 
>>>>> for $ACE in xdmp:directory('opt/MOR/ACE/')/descendant::ACE
>>>>>   [EventSet/GeneralEvent[1]/EventDate gt '2011-03-01T00:00:00']
>>>>>   [EventSet/era:GeneralEvent[1]/EventDate lt '2011-04-01T00:00:00']
>>>>> let $ACEId := $ACE/ACEId
>>>>> let $EventDate := $ACE/EventSet/era:GeneralEvent[1]/era:EventDate
>>>>> return
>>>>> <a>
>>>>> {$ACEId}
>>>>> {$EventDate}
>>>>> </a>
>>>>> 
>>>>> Any ideas are appreciated!
>>>>> 
>>>>> Betty
>>>>> 
>>>>> _______________________________________________
>>>>> General mailing list
>>>>> General at developer.marklogic.com
>>>>> http://developer.marklogic.com/mailman/listinfo/general
>>>> _______________________________________________
>>>> General mailing list
>>>> General at developer.marklogic.com
>>>> http://developer.marklogic.com/mailman/listinfo/general
>>> _______________________________________________
>>> General mailing list
>>> General at developer.marklogic.com
>>> http://developer.marklogic.com/mailman/listinfo/general
>>> 
>> 
>> 
>> /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
>> Betty Harvey                         | Phone:  410-787-9200  FAX: 9830
>> Electronic Commerce Connection, Inc. |
>> harvey at eccnet.com                    | Washington,DC XML Users Grp
>> URL:  http://www.eccnet.com          | http://www.eccnet.com/xmlug
>> /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\\/\/
>> Member of XML Guild (www.xmlguild.org)
>> _______________________________________________
>> General mailing list
>> General at developer.marklogic.com
>> http://developer.marklogic.com/mailman/listinfo/general
>> 
> 
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> http://developer.marklogic.com/mailman/listinfo/general


http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
					


More information about the General mailing list