[MarkLogic Dev General] General question on bi-temporal data

Adrian Creegan adrian.creegan at blanwhite.com
Mon Mar 9 07:27:03 PDT 2015

Hi all,
I am new to MarkLogic so apologies if this is in the wrong forum or if I have a fundamental misunderstanding of this feature.

At the moment I am evaluating MarkLogic as a data store for financial market data.  
Generally the records are end-of-day positions keyed on an instrument identifier with the most recent position being active until a new one is received.
I think something like this use case is covered in the Temporal Developer's Guide with the Valid End date being infinity.
In this scenario, the resulting document splits have valid start and end dates updated so that they form a continuum up to infinity with only one version valid at any point in time.

day-1 { validStart: 'day-1', validEnd: 'infinity' } @t1
day-3 { validStart: 'day-3', validEnd: 'infinity' } @t2

day-1 { systemStart: 't2', systemEnd: 'infinity', validStart: 'day-1', validEnd: 'day-3' }
day-1 { systemStart: 't1', systemEnd: 't2', validStart: 'day-3', validEnd: 'infinity' }
current - day-3 { systemStart: 't2', systemEnd: 'infinity', validStart: 'day-3', validEnd: 'inifinty' }

So if I query using system time between t1 and t2 I will get just the single day-1 record to infinity and if I query on system time t2 or after I get a time-series of 2 values (as expected).

However, if a document is inserted out of time order, it does get inserted into what I would consider to be the valid time continuum:

day-2 { validStart: 'day-2', validEnd: 'infinity' } @t3

day-3 { systemStart: 't2', systemEnd: 't3', validStart: 'day-3', validEnd: 'infinity' }
day-1 { systemStart: 't3', systemEnd: 'infinity', validStart: 'day-1', validEnd: 'day-2' }
day-1 { systemStart: 't2', systemEnd: 't3', validStart: 'day-1', validEnd: 'day-3' }
day-1 { systemStart: 't1', systemEnd: 't2', validStart: 'day-1', validEnd: 'infinity' }

current - day-2 { systemStart: 't3', systemEnd: 'infinity', validStart: 'day-2', validEnd: 'infinity' }

Day 2 for us is not the most recent data in a transactional sense (we insert it at validEnd infinity because we don't know whether there is day 3 data in the store unless we query for it), it should be Day 3.
And if we query for all records at the following system times:
@t1 we get 1 (day-1 - expected)
@t2 we get 2 (day-1 & day-3 - expected)
@t3 we get 2 (day-1 & day-2 - not expected - we need to have day-1, day-2 and day-3). 

This is a problem for us as we cannot guarantee that we will receive feed data on a given day and we can't guarantee that a position will be in a data feed on successive days.
Normally we have to proceed with previous values and back-flush the data if and when we get it in order to fill in the time-series.
Sometimes, especially when a system is being commissioned, we also need to back flush historic data while we are processing current end of day data (again to build up time-series).
And there is also a similar use case when adjustments are retroactively applied to data, for example at the close of regulatory reporting periods.

So I suppose my questions are:

Does MarkLogic allow for back-flushing data in the use cases described above (delayed feed back-flushing, back-flushing historic data and end-of-period adjustments) ?
Are there restrictions on the order of data insertion when trying to maintain a time continuum as above or can it be done ?
Or is my conceptual model of bi-temporal completely wrong and do I need to approach this differently ?

Thanking you in advance,

More information about the General mailing list