lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonik Seeley" <ysee...@gmail.com>
Subject Re: cannot allocate memory for snapshooter
Date Mon, 05 Jan 2009 01:55:36 GMT
On Sun, Jan 4, 2009 at 8:07 PM, Mark Miller <markrmiller@gmail.com> wrote:
> Forking for a small script on something that can have such a large memory
> footprint is just a huge waste of resources. Ideally you might have a tiny
> program running, listening on a socket or something, and it can be alerted
> and do the actual fork (being small itself). Or some other such workaround,
> other than copying a few gig into RAM or swap :)

Well, fork doesn't actually copy anymore (for a long time now) - it's
really only the page tables that get copied and set to copy-on-write
so the fork is actually pretty lightweight.
The issue is that the OS is being conservative and checking that there
would be enough RAM+SWAP available if all of the process address space
did have to be copied/allocated (older versions of linux didn't do
this check and allowed memory overcommit).  The OS doesn't know that
the fork will be followed by an exec.

So the workaround of creating more swap is just so that this OS memory
overcommit check passes.  The swap won't actually be used by the fork
+ exec.

The real fix would be for the JVM to use something like vfork when available.

-Yonik

Mime
View raw message