[MarkLogic Dev General] Apparent Memory Leak in Profiler

Eliot Kimber ekimber at contrext.com
Fri Sep 1 08:15:46 PDT 2017


I can verify that ML 8.07 resolves the memory leak in the profiler. I can now profile 100s of 1000s of tasks no problem.

Cheers,

E.

--
Eliot Kimber
http://contrext.com
 


On 8/28/17, 1:41 PM, "general-bounces at developer.marklogic.com on behalf of Eliot Kimber" <general-bounces at developer.marklogic.com on behalf of ekimber at contrext.com> wrote:

    Thanks—I should be able to test with latest ML 8 in a couple of days.
    
    Cheers,
    
    E.
    
    --
    Eliot Kimber
    http://contrext.com
     
    
    
    On 8/28/17, 12:37 PM, "general-bounces at developer.marklogic.com on behalf of Christopher Hamlin" <general-bounces at developer.marklogic.com on behalf of cbhamlin at gmail.com> wrote:
    
        There was a bug where, under certain circumstances, the profiler will
        result in a query deadlock &/or a resource leak (#45569).  It could be
        that this is what you are seeing.
        
        It was noticed in 8.0-2 and is fixed in the latest release (8.0-7).
        
        On Mon, Aug 28, 2017 at 1:11 PM, Eliot Kimber <ekimber at contrext.com> wrote:
        > I reported earlier that my profiling application was causing MarkLogic to restart after handling about 20,000 tasks. Turns out it was an out-of-memory issue on the server itself (currently configured with 256GB of RAM). We could see a distinct spike in memory usage, at which point the server restarted MarkLogic. I tried different input data sets so it doesn’t appear to be an issue with a particular input document (my data set has a few outliers that are much larger than typical but only a few).
        >
        > Subsequent testing determined that it was the use of the MarkLogic profiler that was causing the memory spike: if I turned off the profiler then memory usage was flat and all the tasks completed as expected.
        >
        > This is ML 8.03. I’m still working on getting my server upgraded to a newer version of MarkLogic so I can see if this is an issue that has already been fixed.
        >
        > So it looks like there’s some kind of memory leak related to the profiler and I’d like to understand what that issue and either understand how to avoid it or report it formally.
        >
        > If it’s a general potential problem with large-scale processing would like to understand how to avoid it or plan for it. If it’s a problem specific to the profiler then need to report it formally and provide appropriate diagnostics.
        >
        > So my questions:
        >
        > 1. Is this a known issue with profiling? I’m guessing not in that I’m probably doing something out-of-the-ordinary vis-à-vis profiling and is something that nobody would see in typical single-instance ad-hoc profiling.
        > 2. What types of MarkLogic processing would cause this kind of memory spike that lasts across the execution of multiple tasks? I would expect the memory required for a given task to be released as soon as the task is complete so I’m guessing it must be an issue with caches or something?
        >
        > Thanks,
        >
        > Eliot
        > --
        > Eliot Kimber
        > http://contrext.com
        >
        >
        >
        >
        > _______________________________________________
        > General mailing list
        > General at developer.marklogic.com
        > Manage your subscription at:
        > http://developer.marklogic.com/mailman/listinfo/general
        _______________________________________________
        General mailing list
        General at developer.marklogic.com
        Manage your subscription at: 
        http://developer.marklogic.com/mailman/listinfo/general
        
    
    
    _______________________________________________
    General mailing list
    General at developer.marklogic.com
    Manage your subscription at: 
    http://developer.marklogic.com/mailman/listinfo/general
    




More information about the General mailing list