[MarkLogic Dev General] Rsync-Like DB Contents Comparison and Update?

Will Thompson wthompson at oconnors.com
Mon Jan 29 07:22:04 PST 2018


I'm not aware of anything out of the box, but when I have use timestamps for similar push/pull remote-sync applications. If every update is timestamped, then you can index that value on all servers, get the cts:max() before a sync, and return everything updated after that timestamp on the master db.

-Will


> On Jan 27, 2018, at 12:13 PM, Eliot Kimber <ekimber at contrext.com> wrote:
> 
> ML 9
> 
> I have a system of servers where a master server gets new remote servers allocated it more or less randomly and dynamically.
> 
> The remote servers need to have a correct copy of a databse on the master server but the database is pretty big (the previously-mentioned 380K doc, 3GB database).
> 
> I can of course sync it with FlexRep but when a new server comes available I don't know what the current state of its local copy of the database is (if it has one at all) so I'm forced to recreate my master server's replication targets and do a full push, which takes an hour or two. 
> 
> In the case where the remote server already has a copy of the database I would like to be able to compare it's contents to the master's and determine what the deltas are, if any, and only handle those, which usually would only be a few docs out of the total set.
> 
> Does there exist this kind of rsync or git-like comparison mechanism, either out of the box or as a public project?
> 
> I'm thinking of something comparable to what git does, which is create hashes of each file and then comparing hashes. 
> 
> I could do this in XQuery but I suspect something more efficient could be done at the forest level, if one knew what one was doing.
> 
> Thanks,
> 
> Eliot
> --
> Eliot Kimber
> https://urldefense.proofpoint.com/v2/url?u=http-3A__contrext.com&d=DwIGaQ&c=IdrBOxAMwHPzAikPNzltHw&r=_thRNTuzvzYaEDwaA_AfnAe5hN2lWgi6qdluz6ApLYI&m=BeNs3-LiGvcHA2blo0b1mD2IhZWNxmkWNzmjoN1LCqQ&s=hR6sEd_Uj-WYl9czUdX7Ir786rHL21bVrxh1JD1rrII&e=
> 
> 
> 
> 
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> Manage your subscription at: 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__developer.marklogic.com_mailman_listinfo_general&d=DwIGaQ&c=IdrBOxAMwHPzAikPNzltHw&r=_thRNTuzvzYaEDwaA_AfnAe5hN2lWgi6qdluz6ApLYI&m=BeNs3-LiGvcHA2blo0b1mD2IhZWNxmkWNzmjoN1LCqQ&s=qGzG2XuYJ_bLuvpvfBkio8StQINqwGZiD0mTRS7_MY0&e=



More information about the General mailing list