[MarkLogic Dev General] xqsync / cpf incompatibility

[Mike Sokolov]

(
I had sent the following report to support at marklogic.com; they indicated that since xqsync is unsupported this mailing list would be a better place to discuss the problem.  Currently I don't have an issue, since I have a good workaround, but I thought others might be interested to know about this, so for posterity's sake:
)


---

I don't know if you would consider this a bug, strictly speaking, but it is some really weird and annoying behavior that cost me a day to track 
down and deal with, so it's probably worth at least a note in the documentation somewhere, or at the very least preservation on this list.

I was having a problem with a cpf pipeline: it would only operate on about 1/4-1/3 of the documents I was loading.  The number fluctuated unpredictably each time I would try loading the same documents, and it didn't seem to have anything to do with the  documents themselves, since different docs would succeed / fail each time.After turning on tracing, I noticed in the logs a variety of different messages: something about checking a transition and skipping a document that sounded suspicious, and then a bunch of deadlocks detected as well.

Anyway, I'll spare you all the detective work, but in the end I tracked this down to xqsync - apparently the way it is setting document properties during loading is incompatible with cpf.  When I disabled its property-loading behavior, everything worked fine.

My suspicion is that xqsync is updating properties after the document is initially created and causing a race condition with cpf, which is doing the same thing.  I'd suggest changing xqsync, if possible, to set the properties during the document creation, or possibly wrapping a transaction around both operations in some way.  If that can't be done then perhaps it's just best to document this in the README.

-Mike