[MarkLogic Dev General] SVC-MAPBIG: Mapped file too large to map: 5825151760 bytes error

[John Zhong]

Thank you, Mike. And yes, I knew the server was overloaded. I just want to
know if there is another suggestion except upgrading the server. It is
probably no.

Thanks again.
John

On Fri, Nov 4, 2011 at 11:00 PM, Michael Blakeley <mike at blakeley.com> wrote:

> To be blunt: that host is out of memory and over capacity.
>
> Stop ingestion right away, and revert the configuration changes you have
> made. Turning off merges is almost always a mistake: there ought to be a
> pop quiz built into the configuration screen. You can easily paint yourself
> into a corner. The other changes were almost certainly counterproductive,
> too.
>
> How can I tell that you are out of memory? The error message states that
> MarkLogic was unable to map a 5-GiB index. That is only one of many indexes
> that need to memory-map, so I would guess that the system is badly
> overloaded. The rest of the data confirms it: on a 16-GiB host your total
> index memory (aka forest memory) is about 12-GiB. You can't use all the
> host memory for forest indexes: you'll need about that much again for
> group-level caches, and about 25% of RAM for queries, merges, and the OS.
> For your data so far, you ought to have at least 32-GB RAM.
>
> Do you know where you are in your total load? Are you about 10% done, or
> closer to 75% done?
>
> With the system in its current state, it will probably use a lot of
> virtual memory for paging, and performance will probably be unacceptable.
> Am I right in thinking this is a VM? If so, you may be able to upgrade the
> VM memory without much trouble. Virtual or not, if a memory upgrade isn't
> practical then I would recommend clearing the database and starting over,
> and this time stop before you are out of memory.
>
> -- Mike
>
> On 4 Nov 2011, at 00:33 , John Zhong wrote:
>
> > Hi all,
> >
> > I am running MarkLogic 4.1-6 version and encountered the below error
> today:
> >
> > XDMP-FORESTERR: Error in merge of forest: SVC-MAPBIG: Mapped file too
> large to map: 5825151760 bytes: 'D:\Program
> Files\MarkLogic\Data\Forests\forest1\00001cde\580f5ba44789ea78_dateTime'
> >
> > The MarkLogic instance is running on a EC2 server has 1 CUP with 2
> cores, 17.1GB RAM, 64 bit windows server 2008. We have created only one
> database with one forest, now the forest has 6 stands, total is 86GB, and
> there is still 300GB disk space available.
> >
> >                active fragments      deleted fragments      on disk size
>    in memory size
> > stand 1:     2,922,206               3,320,621                 36,246 MB
>      4,162 MB
> > stand 2:     1,111,868               1,795,830                 39,987 MB
>      6,836 MB
> > stand 3:         63,545                  107,616                   3,588
> MB         606 MB
> > stand 4:           7,620                   24,492
>  925 MB         163 MB
> > stand 5:           5,638                     6,882
>  479 MB          79 MB
> > stand 6:         11,757                    28,626
> 1,761 MB         319 MB
> > TOTAL:     4,122,634               5,284,067                  82,986 MB
>     12,165 MB
> >
> > We enable the default merge setting and tried to modify the merge max
> size to 2048 then merge manually, but still failed.
> >
> > Now, we disable the merge.
> >
> > Any sugestion will be very appreciate!!
> >
> > Thank you in advance.
> > John
> >
> >
> > _______________________________________________
> > General mailing list
> > General at developer.marklogic.com
> > http://developer.marklogic.com/mailman/listinfo/general
>
> _______________________________________________
> General mailing list
> General at developer.marklogic.com
> http://developer.marklogic.com/mailman/listinfo/general
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://developer.marklogic.com/pipermail/general/attachments/20111104/30df7374/attachment.html