[MarkLogic Dev General] Reindexing estimation

Danny Sinang d.sinang at gmail.com
Wed May 22 13:20:05 PDT 2013


Hi Michael,

Does this behavior (of queries using the old indexes while reindexing is
going on) apply :

1. to manually invoked reindexing - not just to reindexing caused by an ML
upgrade ?
2. whether or not "index detection" is set to "automatic" or "none" ?

Regards,
Danny


---------- Forwarded message ----------
From: Michael Blakeley <mike at blakeley.com>
Date: Tue, May 14, 2013 at 4:19 PM
Subject: Re: [MarkLogic Dev General] Reindexing estimation
To: MarkLogic Developer Discussion <general at developer.marklogic.com>


The database is designed so that indexes are versioned and
forward-compatible. Thus reindexing is a fully online operation. After a
major upgrade the old indexes are still there. The forests want to build
new ones, hence the reindex. But until that happens, queries use whatever
is available.

This passage from http://docs.marklogic.com/admin-help/database might help:

> • format compatibility specifies the version compatibility that MarkLogic
Server applies to the indexes for this database during request evaluation.
Setting this to a value other than automatic specifies that all forest data
has the specified on-disk format, and it disables the automatic checking
for index compatibility information. The automatic detection occurs during
database startup and after any database configuration changes, and can take
some time and system resources for very large forests and for very large
clusters. The default value of automatic is recommended for most
installations.
> • index detection specifies whether to auto-detect index compatibility
between the content and the current database settings. This detection
occurs during database startup and after any database configuration
changes, and can take some time and system resources for very large forests
and for very large clusters. Setting this to none also causes queries to
use the current database index settings, even if some settings have not
completed reindexing. The default value of automatic is recommended for
most installations.


-- Mike

On 14 May 2013, at 09:20 , Danny Sinang <d.sinang at gmail.com> wrote:

> HI Michael,
>
> What do you mean by "Queries will use backward-compatibility mode" ?
>
> During reindexing, queries would be slower since there isn't any
completed index yet, right ?
>
> Also, while reindexing has not completed, will there be any facets
available ?
>
> Regards,
> Danny
>
>
> On Tue, May 14, 2013 at 11:56 AM, Michael Blakeley <mike at blakeley.com>
wrote:
> Probably not, because disk speed is often the bottleneck at that scale
and you haven't said anything about that. But it will take roughly as long
as ingesting all the data would take. My guess is that could be anywhere
from 1-10 days.
>
> If you find that reindexing is bad for query performance, you can disable
it until you are ready. Queries will use backward-compatibility mode, just
like they would  during reindexing anyway. Then you can run reindexing a
little at a time, say on the weekends, until you have caught up. That will
take longer, but may impact operations less.
>
> -- Mike
>
> On May 14, 2013, at 6:04, "Roy Chowdhury, Soumadri" <
srroychowdhury at innodata.com> wrote:
>
>> Hi,
>>
>> I am upgrading from MarkLogic 5.x to 6, which contains ~2 terabytes
data. I believe while upgrading, MarkLogic will reindex all the data. The
database has ~17 range indexes defined. Can anyone help me how to estimate
approximately the time MarkLogic will take to reindex all the data, without
actually starting it and checking the estimation from 'Status' tab of
database?
>>
>> Regards,
>> Soumadri Roy
>>
>>
>> "This e-mail and any attachments transmitted with it are for the sole
use of the intended recipient(s) and may contain confidential , proprietary
or privileged information. If you are not the intended recipient, please
contact the sender by reply e-mail and destroy all copies of the original
message. Any unauthorized review, use, disclosure, dissemination,
forwarding, printing or copying of this e-mail or any action taken in
reliance on this e-mail is strictly prohibited and may be unlawful."
>> _______________________________________________
>> General mailing list
>> General at developer.marklogic.com
>> http://developer.marklogic.com/mailman/listinfo/general
>
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> http://developer.marklogic.com/mailman/listinfo/general
>
>
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> http://developer.marklogic.com/mailman/listinfo/general

_______________________________________________
General mailing list
General at developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://developer.marklogic.com/pipermail/general/attachments/20130522/ef22a246/attachment.html 


More information about the General mailing list