lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: cannot allocate memory for snapshooter
Date Mon, 05 Jan 2009 02:12:41 GMT
Yonik Seeley wrote:
> On Sun, Jan 4, 2009 at 8:07 PM, Mark Miller <markrmiller@gmail.com> wrote:
>   
>> Forking for a small script on something that can have such a large memory
>> footprint is just a huge waste of resources. Ideally you might have a tiny
>> program running, listening on a socket or something, and it can be alerted
>> and do the actual fork (being small itself). Or some other such workaround,
>> other than copying a few gig into RAM or swap :)
>>     
>
> Well, fork doesn't actually copy anymore (for a long time now) - it's
> really only the page tables that get copied and set to copy-on-write
> so the fork is actually pretty lightweight.
>   
Right, copying was the wrong word. It depends. Depending on your Unix 
variant, it will actually use vfork, or sometimes..., or sometimes
sometimes you have no option to share. (Because you can screw with the 
parent, I've seen warnings in the doc for vfork that its not recommended 
even for use - but this could be old now, and was for a particular 
version of UNIX that I don't remember)
> The issue is that the OS is being conservative and checking that there
> would be enough RAM+SWAP available if all of the process address space
> did have to be copied/allocated (older versions of linux didn't do
> this check and allowed memory overcommit).  The OS doesn't know that
> the fork will be followed by an exec.
>   
I don't think you can just count on that in a unix environment. Maybe 
Linux took care of it, but is that common on all versions of Unix? And 
if you have an older version of Linux?
> So the workaround of creating more swap is just so that this OS memory
> overcommit check passes.  The swap won't actually be used by the fork
> + exec.
>   
Again, only if your lucky. It depends on the many implementations of 
fork. A lot of times fork is actually vfork or something, but solr can't 
count on it for everybody I wouldn't think.
> The real fix would be for the JVM to use something like vfork when available.
>   
Which kind of happens under the scenes if your lucky already. Some unix 
guys don't like it, and I assume thats why its not the standard (overly 
concerned with the child process mucking up the parent process).

I shouldn't have said copy - the issue is that we are looking for way to 
much RAM. A JVM using 5 gig will look for another 5 - thats terrible. I 
don't link we can solve it in a universal way for Unix by relying on 
forking the JVM though. Its hit or miss. The real fix can't depend on 
your OS varient and its version I wouldn't think.

Mime
View raw message