Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: zookeeper-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of ted.dunning@gmail.com
 designates 209.85.216.204 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=gbGBMEMglGYE0XRLEFGLw3g8UBhX3Q6Az5DzQJ+sOMDE815F9ZNJZ7aXkoRLjiVU3q
         LLBeSJ+MpXTQhxRxWw7/n8l07Yht0HtQ6vmsT3y+02pSvVUBNq2LRglBGA9NjFuD8gCO
         gYT2Nw9xglqMFlK/EAs0/Cz2lS6A5FAF3r4h4=
MIME-Version: 1.0
In-Reply-To: <4B5DD841.3000709@apache.org>
References: <34e4e27f1001221740r51fdb34fl2b2104ca9000e828@mail.gmail.com>
	<C77F9582.2D57A%mahadev@yahoo-inc.com>
 <34e4e27f1001221801n4c2e9918qd0241866654bec66@mail.gmail.com>
	<4B5DD841.3000709@apache.org>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Mon, 25 Jan 2010 13:14:14 -0800
Message-ID: <c7d45fc71001251314s41686ac3i5b14e9b790e97bbb@mail.gmail.com>
Subject: Re: Server exception when closing session
To: zookeeper-user@hadoop.apache.org
Cc: jscheid@velocetechnologies.com
Content-Type: multipart/alternative; boundary=0016e64c2782edf2a4047e03a74d

--0016e64c2782edf2a4047e03a74d
Content-Type: text/plain; charset=UTF-8

Be very cautious about misdirection here.  It is easy to focus on the ZK
server-side GC's.  In my experience, I have had many more GC related ZK
problems caused by *client* side GC.  If the client checks out for a minute,
you get disconnects and session expiration which is good for debugging that
code, but generally indicates a really bad user experience.  Lots of times
that bad user experience is somewhat covered up by your load balancer and
other general redundancy to you may notice it first from ZK forcing you to
think about these things.

On Mon, Jan 25, 2010 at 9:43 AM, Patrick Hunt <phunt@apache.org> wrote:

> GC and disk IO (transactional log in particular) will cause significant
> latency in some cases. See this for details on the types of things you
> should look at:
>
> http://wiki.apache.org/hadoop/ZooKeeper/Troubleshooting
>
> I've seen cases where the JVM will pause for 2+ minutes for GC, in some
> cases I've seen 4+ and I've heard of worse than that. Tuning GC (in
> particular using incremental/cms gc) is critical for consistently low
> latencies.
>
> Patrick
>
> Josh Scheid wrote:
>
>> On Fri, Jan 22, 2010 at 17:48, Mahadev Konar <mahadev@yahoo-inc.com>
>> wrote:
>>
>>>  The server latency does seem huge. What os and hardware are you running
>>> it
>>> on?
>>>
>>
>> RHEL4 2.8GHz 2-core AMD.  8GB RAM.
>> I will check with my admin for evidence of swap activity, but I don't
>> anticipate any.
>>
>> This is bursty.  I'm currently seeing ~200 connections, but maxlat in
>> the last hour has been 185ms with 4ms avg.
>>
>>  What is usage model of zookeeper?
>>>
>>
>> Distributed lock service.  Using the lock recipe.  Hosts hold a lock
>> for 5s to a couple of minutes with zero to dozens of waiters.
>>
>>  How much memory are you allocating to the server?
>>>
>>
>> That's a good question.  I'm not an expert at java deployment.  I just
>> use zkServer.sh defaults.
>> sun-jre 1.6.0_14.  It's taking 1188MB of virtual memory right now,
>> 100MB resident.
>>
>>  The debug well exacerbate the problem.
>>>
>>
>> OK.
>>
>>  A dedicated disk means the following:
>>> Zookeeper has snapshots and transaction logs. The datadir is the
>>> directory
>>> that stores the transaction logs. Its highly recommended that this
>>> directory
>>> be on a separate disk that isnt being used by any other process. The
>>> snapshots can sit on a disk that is being used by the OS and can be
>>> shared.
>>>
>>
>> Yeah, I understand.  I just need to get that set up and was hoping
>> that my load wouldn't warrant it.
>>
>>  Also, Pat ran some tests for serve lantecies at:
>>>
>>> http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview
>>>
>>> You can take a look at that as well and see what the expected performance
>>> should be for your workload.
>>>
>>
>> I will take a look at that.  Thank you for your time.
>>
>> -Josh
>>
>


-- 
Ted Dunning, CTO
DeepDyve

--0016e64c2782edf2a4047e03a74d--