Naha,

Do you have the distribution of the request latencies? If you're seeing
something like the top 1% of requests taking 5s while the rest take double
digit ms, and average out to three-digit ms... then it's likely that some
requests are still waiting on the snapshot write despite you having
disabled wait for fsync.

If the distribution is not spiky though - then I haven't seen that case.

Best Regards,
Martin Kou


On Thu, Aug 8, 2013 at 11:40 AM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Since this zookeeper cluster is a pre-prod cluster and is used heavily, we
> have turned the forceSync to OFF. So we don't see the fsync warnings on
> this zookeeper cluster.
>
> Thanks,
> Neha
>
>
> On Thu, Aug 8, 2013 at 11:18 AM, Martin Kou <bitanarch@gmail.com> wrote:
>
> > Do you see fsync() warnings in your ZooKeeper log? It could be disk
> writes
> > taking too long.
> >
> > Best Regards,
> > Martin Kou
> >
> >
> > On Thu, Aug 8, 2013 at 10:37 AM, Jun Rao <junrao@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > We have a 3-node ZK cluster running 3.3.4 that are experiencing high
> > > latency. From "stat", we saw avg latency to be 300-400ms consistently.
> > > Outstanding requests jumps to ~600 from time to time. However, the
> > server's
> > > I/O and CPU are both low. If I start a new ZK client and do a few
> writes,
> > > the observed response time in the client is also low (1ms).
> > >
> > > What's the best way to figure out why the reported latency in ZK server
> > is
> > > high? Also, are  min/avg/max latency computed on a rolling time window?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> >
>