zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: setting zookeeper heap size
Date Fri, 11 Nov 2011 17:33:37 GMT
I agree. I've been meaning to create a set of jiras for this, all
related to resource availability/env, it's on my todo list (came up
recently at a customer site). I see a number of things:

1) heap:
a) sane default for max mem (easy), plus docs and such.
b) turn on CMS/parallelGC by default
c) verify that the server correctly handles OOM in all cases (hard)
2) verify that there are sufficient FDs available (ie ulimit - also
came up recently at customer site)
a) warn if too low
b) try to set to some sane default, perhaps 2k unless already set higher
c) we might want to have the server not accept client connections
after some fd limit is reached. This would protect the storage system
from failure.
3) also see ZOOKEEPER-1239
a) in addition to fsync stall we should have a check for GC stop the
world pauses.

HBase seems to do all these things (e.g. see 1239). We might copy what
they are doing.

These are also all pretty easy things for new ZK contributors to take
on. (hint hint). Esp ones that are appreciative when experienced
contributors quickly takeup/fix the particularly nasty bugs. (hint
hint) ;-)

Patrick

On Fri, Nov 11, 2011 at 6:41 AM, Camille Fournier <camille@apache.org> wrote:
> The problem in my mind is that it raises the spectre of users
> overcommitting their servers accidentally, and instead of seeing the
> result as a hard fail if their VM runs out of available memory, they
> will see it as a useless system when everything starts swapping to
> disk because the total heap size for VMs on the machine has grown
> beyond the reasonably allocatable physical memory of the machine.
> I don't really care about the ergonomics of client VMs, but server VMs
> in 64bit servers defaulting to 1/4 the memory of the system seems
> excessive. Anyway, whether it is reasonable for the vast case of VMs
> out there in the wild or not, it is way overboard for the vast case of
> ZooKeepers out there in the wild. I think we should consider offering
> guidance on this in documentation or the scripts.
>
> C
>
> On Thu, Nov 10, 2011 at 11:59 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
>> The behavior without xmx is actually pretty reasonable.  Note that the
>> default of half is only up to a few hundred MB of memory (not GB).  Even
>> when lots of memory is allocated, the jvm avoids expanding the actual use
>> beyond reasonable need.
>>
>> On Thu, Nov 10, 2011 at 8:35 PM, Camille Fournier <camille@apache.org>wrote:
>>
>>> Btw I'm pretty sure res is used heap and virt is just mapped but yet to be
>>> filled. Use jstat -gc to see actual usage for the various collectors.
>>>
>>> Also, if it's true that the jvm defaults to a xmx of half the ram (wtf bad
>>> decision) we had better ship zk start scripts with more reasonable xmx
>>> configured.
>>> Although I've used u18 and u23 extensively on 64bit servers and never seen
>>> it default to half the physical memory, or 32gig.
>>>
>>> C
>>>
>>> From my phone
>>> On Nov 10, 2011 6:50 PM, "Ed Sexton" <sextonhadoop@gmail.com> wrote:
>>>
>>> > Hi Henry,
>>> >
>>> > I checked the process listing via ps and top, and saw VIRT=6477m  and
>>> > RES=83m.
>>> >
>>> > How would one check if it is the jvm heap vs mapped space?
>>> >
>>> > Thanks as always Henry.
>>> > Ed
>>> >
>>> > On Thu, Nov 10, 2011 at 2:04 PM, Henry Robinson <henry@cloudera.com>
>>> > wrote:
>>> >
>>> > > Ed - how are you making the observation that the heap size is 6477m?
Is
>>> > it
>>> > > from a profiler, or from top or similar? The amount of virtual memory
>>> > > mapped for Java processes can often be very large, especially on recent
>>> > > RHEL systems with the new arena allocator in glibc. Worth checking
>>> > whether
>>> > > it's the JVM heap or the amount of mapped space.
>>> > >
>>> > > cheers,
>>> > > Henry
>>> > >
>>> > > On 10 November 2011 13:47, Ed Sexton <sextonhadoop@gmail.com>
wrote:
>>> > >
>>> > > > Hello Group-
>>> > > >
>>> > > > I am running CDH zookeeper version 3.3.3+12.12 and noticed that
>>> > zookeeper
>>> > > > is running with a heapsize of 6477m.
>>> > > >
>>> > > > I start zookeeper standalone, via
>>> /etc/init.d/hadoop-zookeeper-server.
>>> >  I
>>> > > > do not see reference to any ZOOKEEPER_HEAPIZE in the startup or
>>> > zoo.cfg,
>>> > > > nor do I see any special JVMFLAGS in export/usr/bin/zookeeper-server,
>>> > > which
>>> > > > is what is called by the startup script, it just has this definition:
>>> > > >
>>> > > > JVMFLAGS=-Dzookeeper.log.threshold=INFO
>>> > > >
>>> > > > Can someone guide me to where the 6GB heapsize is being referenced?
>>> >  This
>>> > > > seems a bit large of a heap.
>>> > > >
>>> > > > Thanks for your guidance.
>>> > > >
>>> > > > Sincerely,
>>> > > > Ed
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Henry Robinson
>>> > > Software Engineer
>>> > > Cloudera
>>> > > 415-994-6679
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Sincerely,
>>> > Ed Sexton
>>> > e: sextonhadoop@gmail.com
>>> > gv: (408) 475-2358
>>> >
>>>
>>
>

Mime
View raw message