jackrabbit-oak-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jukka Zitting <jukka.zitt...@gmail.com>
Subject Re: Segment Store and unbounded instances of MappedByteBuffer causing high RAM usage
Date Tue, 19 Nov 2013 17:59:03 GMT
Hi,

On Tue, Nov 19, 2013 at 1:57 AM, Chetan Mehrotra
<chetan.mehrotra@gmail.com> wrote:
> On some systems running on Oak (on Windows) where lots of nodes get
> created we are seeing high RAM usage. The Java process memory (shown
> directly under task manager. See note on Working Set below) remains
> within expected range but system starts paging memory quite a bit and
> slows down.
>
> On checking the runtime state following things can be observed
> * FileStore has 186 TarFile and each TarFile refers to 256MB
> MappedByteBuffer taking around 46 GB of memory
> * The JVM process constantly shows 50% CPU usage. Checking further
> indictes that AsynchUpdate is running and Lucene indexing is being
> performed
> * Windows Task Manager shows 7 .66 GB (total 8 GB) overall memory
> usage with Java process only showing 1.4GB usage. No other process
> account for so high usage
> * Checking the Working Set [1] shows 7 GB memory being used by Java
> process alone
>
> From my basic understanding of the Memory Mapped file usage it should
> not cause msuch resource crunch and OS would be able to manage the
> memory. However probably in Oak case all these TarFiles are getting
> accessed frequently which leads to these memory mapped pages held in
> physical memory.

Hmm, I wonder if we do have a problem here or if the system is just
working as designed. If you have 8GB of memory on the system and no
other processes using much of it, then it would be natural and
expected for the operating system to use most of the available memory
for caching the memory mapped files that are actively being accessed
by TarMK.

The 50% CPU usage you mention suggests that the system isn't actually
being blocked by the paging behavior. Instead it's just busy crunching
through the content, which causes newly accessed data to be paged in
and old to be discarded. The same disk IO would be required without
memory mapping. If thrashing was occurring, you'd expect to see much
lower CPU usage.

> Should we put upper cap on number of TarFiles opened?

If this indeed is a problem, a better solution would probably be to
limit the maximum working set size at the operating system level.
Alternatively we could switch to using normal file IO for bulk
segments (ones that only contain parts of large binaries).

> If any other data needs to be collected to determine the problem cause
> further then let me know?

Can you reduce the behavior to a standalone test case? Or you might
want to try running the system with a 32 bit JVM, which should force
TarMK to use normal file IO with a fixed-size buffer instead of memory
mapping.

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/ms686237(v=vs.85).aspx

BR,

Jukka Zitting

Mime
View raw message