[MarkLogic Dev General] General question on bi-temporal data

Michael Blakeley mike at blakeley.com
Wed Mar 11 19:16:37 PDT 2015


Temporal documents are new to me too, so I hope I won't steer you off-course.

What does your document-level data model look like? That is, what goes into each document when you call document-insert?

I'd expect each document to model a single instrument and the price data for that instrument. But it sounds like your model could be different: if you're trying to insert data for multiple instruments in one document, that might be part of the problem.

Are you using 'infinity' as a lexical value for validEnd? That could be part of the problem too. The docs examples show systemStart and systemEnd as null, and say that will be mapped to infinity. There are also examples showing validEnd as 9999-12-31T23:59:59.99Z, and there's a note that "The date-time 9999-12-31T23:59:59 represents infinity and is used when there is no valid date-time yet for that start or end time."

-- Mike

> On 9 Mar 2015, at 07:27 , Adrian Creegan <adrian.creegan at blanwhite.com> wrote:
> 
> Hi all,
> I am new to MarkLogic so apologies if this is in the wrong forum or if I have a fundamental misunderstanding of this feature.
> 
> 
> At the moment I am evaluating MarkLogic as a data store for financial market data.  
> Generally the records are end-of-day positions keyed on an instrument identifier with the most recent position being active until a new one is received.
> I think something like this use case is covered in the Temporal Developer's Guide with the Valid End date being infinity.
> In this scenario, the resulting document splits have valid start and end dates updated so that they form a continuum up to infinity with only one version valid at any point in time.
> 
> 
> e.g.
> Add:
> day-1 { validStart: 'day-1', validEnd: 'infinity' } @t1
> day-3 { validStart: 'day-3', validEnd: 'infinity' } @t2
> 
> 
> Becomes:
> day-1 { systemStart: 't2', systemEnd: 'infinity', validStart: 'day-1', validEnd: 'day-3' }
> day-1 { systemStart: 't1', systemEnd: 't2', validStart: 'day-3', validEnd: 'infinity' }
> current - day-3 { systemStart: 't2', systemEnd: 'infinity', validStart: 'day-3', validEnd: 'inifinty' }
> 
> 
> So if I query using system time between t1 and t2 I will get just the single day-1 record to infinity and if I query on system time t2 or after I get a time-series of 2 values (as expected).
> 
> 
> However, if a document is inserted out of time order, it does get inserted into what I would consider to be the valid time continuum:
> 
> 
> Add:
> day-2 { validStart: 'day-2', validEnd: 'infinity' } @t3
> 
> 
> Becomes:
> day-3 { systemStart: 't2', systemEnd: 't3', validStart: 'day-3', validEnd: 'infinity' }
> day-1 { systemStart: 't3', systemEnd: 'infinity', validStart: 'day-1', validEnd: 'day-2' }
> day-1 { systemStart: 't2', systemEnd: 't3', validStart: 'day-1', validEnd: 'day-3' }
> day-1 { systemStart: 't1', systemEnd: 't2', validStart: 'day-1', validEnd: 'infinity' }
> 
> 
> current - day-2 { systemStart: 't3', systemEnd: 'infinity', validStart: 'day-2', validEnd: 'infinity' }
> 
> 
> Day 2 for us is not the most recent data in a transactional sense (we insert it at validEnd infinity because we don't know whether there is day 3 data in the store unless we query for it), it should be Day 3.
> And if we query for all records at the following system times:
> @t1 we get 1 (day-1 - expected)
> @t2 we get 2 (day-1 & day-3 - expected)
> @t3 we get 2 (day-1 & day-2 - not expected - we need to have day-1, day-2 and day-3). 
> 
> 
> This is a problem for us as we cannot guarantee that we will receive feed data on a given day and we can't guarantee that a position will be in a data feed on successive days.
> Normally we have to proceed with previous values and back-flush the data if and when we get it in order to fill in the time-series.
> Sometimes, especially when a system is being commissioned, we also need to back flush historic data while we are processing current end of day data (again to build up time-series).
> And there is also a similar use case when adjustments are retroactively applied to data, for example at the close of regulatory reporting periods.
> 
> 
> So I suppose my questions are:
> 
> 
> Does MarkLogic allow for back-flushing data in the use cases described above (delayed feed back-flushing, back-flushing historic data and end-of-period adjustments) ?
> Are there restrictions on the order of data insertion when trying to maintain a time continuum as above or can it be done ?
> Or is my conceptual model of bi-temporal completely wrong and do I need to approach this differently ?
> 
> 
> Thanking you in advance,
> Adrian.
> 
> 
> 
> 
> 
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> http://developer.marklogic.com/mailman/listinfo/general



More information about the General mailing list