From user-return-6644-apmail-zookeeper-user-archive=zookeeper.apache.org@zookeeper.apache.org Fri Aug 9 09:31:50 2013 Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3A98010994 for ; Fri, 9 Aug 2013 09:31:50 +0000 (UTC) Received: (qmail 36711 invoked by uid 500); 9 Aug 2013 09:31:49 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 36286 invoked by uid 500); 9 Aug 2013 09:31:41 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 36242 invoked by uid 99); 9 Aug 2013 09:31:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Aug 2013 09:31:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of bitanarch@gmail.com designates 209.85.223.179 as permitted sender) Received: from [209.85.223.179] (HELO mail-ie0-f179.google.com) (209.85.223.179) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Aug 2013 09:31:33 +0000 Received: by mail-ie0-f179.google.com with SMTP id c11so3853760ieb.10 for ; Fri, 09 Aug 2013 02:31:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=vXrxRZ0kq1hC3yeolvOcVQ/4gExXYm66Bijgq6MJboQ=; b=uyDY9YJwwa1RkHP9PmVc9WFgM8CfnbuErHzh1Z7d3eLmcoec4zNGmLU5GANWDq/fKG nn55J7OM6u8XE6s8pybotQI6mDYupwMXJOL2CI+jTN8m6FDuwq+ykDlTfc7udKG3Ww7m d7lBP+kwQUFuMSsnRzpupRa1ZeK99rCmirjHsVt7IyirqeMNk7zLJP2voB1n9J5urijA /plcDwLTNpOO7prKjWlBMtajSw9GqrivqtOt6J7v9cITfINVU2D9/av+qlZD98F4reLw 0mi4twvzejxl5iB4xojyueFkrP1NzKWwSEA69+e4wx8C3ZCCVnd/dDa0bR1NHodZ/08O C/Kg== MIME-Version: 1.0 X-Received: by 10.50.154.106 with SMTP id vn10mr1747991igb.0.1376040673072; Fri, 09 Aug 2013 02:31:13 -0700 (PDT) Received: by 10.50.116.194 with HTTP; Fri, 9 Aug 2013 02:31:12 -0700 (PDT) In-Reply-To: References: Date: Fri, 9 Aug 2013 02:31:12 -0700 Message-ID: Subject: Re: determining ZK latency issues From: Martin Kou To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary=047d7bd74b6283dd2804e3806f88 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bd74b6283dd2804e3806f88 Content-Type: text/plain; charset=ISO-8859-1 Naha, Do you have the distribution of the request latencies? If you're seeing something like the top 1% of requests taking 5s while the rest take double digit ms, and average out to three-digit ms... then it's likely that some requests are still waiting on the snapshot write despite you having disabled wait for fsync. If the distribution is not spiky though - then I haven't seen that case. Best Regards, Martin Kou On Thu, Aug 8, 2013 at 11:40 AM, Neha Narkhede wrote: > Since this zookeeper cluster is a pre-prod cluster and is used heavily, we > have turned the forceSync to OFF. So we don't see the fsync warnings on > this zookeeper cluster. > > Thanks, > Neha > > > On Thu, Aug 8, 2013 at 11:18 AM, Martin Kou wrote: > > > Do you see fsync() warnings in your ZooKeeper log? It could be disk > writes > > taking too long. > > > > Best Regards, > > Martin Kou > > > > > > On Thu, Aug 8, 2013 at 10:37 AM, Jun Rao wrote: > > > > > Hi, > > > > > > We have a 3-node ZK cluster running 3.3.4 that are experiencing high > > > latency. From "stat", we saw avg latency to be 300-400ms consistently. > > > Outstanding requests jumps to ~600 from time to time. However, the > > server's > > > I/O and CPU are both low. If I start a new ZK client and do a few > writes, > > > the observed response time in the client is also low (1ms). > > > > > > What's the best way to figure out why the reported latency in ZK server > > is > > > high? Also, are min/avg/max latency computed on a rolling time window? > > > > > > Thanks, > > > > > > Jun > > > > > > --047d7bd74b6283dd2804e3806f88--