[MarkLogic Dev General] my link resolver is slow

Michael Sokolov sokolov at ifactory.com
Thu Aug 9 04:48:14 PDT 2012


On 8/8/2012 7:14 PM, Geert Josten wrote:
> Another two cents:
>
> Keeping track where you have been and where not is not trivial. I
> struggled with something similar too recently, though in your case link
> targets could (in theory at least) disappear, not in mine. Do you take
> that into account too? Means you may need to revisit existing links once
> in a while to recheck them. In my case I could keep track with an extra
> attribute (or something like that), you on the other hand may need to
> store a timestamp for each link, and schedule recheck processes..
Yes this is a perennial problem.  I am not worrying about targets 
disappearing: this content is cumulative.  At the moment I am using a 
document-level timestamp to avoid revisiting documents checked during 
this run.  Another thought we had was to use an extra attribute on the 
links, yes, but I don't think we need that yet.
>
> You do know you can pass in multiple element names into an
> element-attribute-value-query, right? Just to be sure.. ;-P
Yes, that's what we're doing.  I thought this would be "free", but in at 
least one measurement we saw a noticeable slowdown when we searched for 
many element/attribute combos this way, as compared to only a single 
one.  So one optimization we're considering is to restrict this set in 
certain cases where we know only those elements can appear.

-Mike


More information about the General mailing list