[MarkLogic Dev General] General question on bi-temporal data

Fei Xue Fei.Xue at marklogic.com
Thu Mar 12 11:51:09 PDT 2015

Hi Adrian,

As Anthony mentioned, it is totally fine to insert historical entries. In fact, it is one of the ways how bitemporal is useful for this use case.

In your case when you are doing so, because you are making an update to a historical range, then using validEnd of INF is not suitable.

So instead of 

day-2 { validStart: 'day-2', validEnd: 'infinity' } @t3  => this will overwrite what you already know about for day -3

you should add:

day-2 { validStart: 'day-2', validEnd: 'day-3' } @t3

=> this will overwrite info for day-2 => day-3 and the info you inserted @t2 will not be affected

Hope this makes sense?


-----Original Message-----
From: general-bounces at developer.marklogic.com [mailto:general-bounces at developer.marklogic.com] On Behalf Of Adrian Creegan
Sent: Monday, March 09, 2015 7:27 AM
To: general at developer.marklogic.com
Subject: [MarkLogic Dev General] General question on bi-temporal data

Hi all,
I am new to MarkLogic so apologies if this is in the wrong forum or if I have a fundamental misunderstanding of this feature.

At the moment I am evaluating MarkLogic as a data store for financial market data. Generally the records are end-of-day positions keyed on an instrument identifier with the most recent position being active until a new one is received.
I think something like this use case is covered in the Temporal Developer's Guide with the Valid End date being infinity.
In this scenario, the resulting document splits have valid start and end dates updated so that they form a continuum up to infinity with only one version valid at any point in time.

day-1 { validStart: 'day-1', validEnd: 'infinity' } @t1
day-3 { validStart: 'day-3', validEnd: 'infinity' } @t2

day-1 { systemStart: 't2', systemEnd: 'infinity', validStart: 'day-1', validEnd: 'day-3' }
day-1 { systemStart: 't1', systemEnd: 't2', validStart: 'day-3', validEnd: 'infinity' } current - day-3 { systemStart: 't2', systemEnd: 'infinity', validStart: 'day-3', validEnd: 'inifinty' }

So if I query using system time between t1 and t2 I will get just the single day-1 record to infinity and if I query on system time t2 or after I get a time-series of 2 values (as expected).

However, if a document is inserted out of time order, it does get inserted into what I would consider to be the valid time continuum:

day-2 { validStart: 'day-2', validEnd: 'infinity' } @t3

day-3 { systemStart: 't2', systemEnd: 't3', validStart: 'day-3', validEnd: 'infinity' }
day-1 { systemStart: 't3', systemEnd: 'infinity', validStart: 'day-1', validEnd: 'day-2' }
day-1 { systemStart: 't2', systemEnd: 't3', validStart: 'day-1', validEnd: 'day-3' }
day-1 { systemStart: 't1', systemEnd: 't2', validStart: 'day-1', validEnd: 'infinity' }

current - day-2 { systemStart: 't3', systemEnd: 'infinity', validStart: 'day-2', validEnd: 'infinity' }

Day 2 for us is not the most recent data in a transactional sense (we insert it at validEnd infinity because we don't know whether there is day 3 data in the store unless we query for it), it should be Day 3.
And if we query for all records at the following system times:
@t1 we get 1 (day-1 - expected)
@t2 we get 2 (day-1 & day-3 - expected)
@t3 we get 2 (day-1 & day-2 - not expected - we need to have day-1, day-2 and day-3). 

This is a problem for us as we cannot guarantee that we will receive feed data on a given day and we can't guarantee that a position will be in a data feed on successive days.
Normally we have to proceed with previous values and back-flush the data if and when we get it in order to fill in the time-series.
Sometimes, especially when a system is being commissioned, we also need to back flush historic data while we are processing current end of day data (again to build up time-series).
And there is also a similar use case when adjustments are retroactively applied to data, for example at the close of regulatory reporting periods.

So I suppose my questions are:

Does MarkLogic allow for back-flushing data in the use cases described above (delayed feed back-flushing, back-flushing historic data and end-of-period adjustments) ?
Are there restrictions on the order of data insertion when trying to maintain a time continuum as above or can it be done ?
Or is my conceptual model of bi-temporal completely wrong and do I need to approach this differently ?

Thanking you in advance,

General mailing list
General at developer.marklogic.com

More information about the General mailing list