Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of tyler@datastax.com designates
 209.85.217.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <4FF79AE3.9010302@syncopated.net>
References: <4FF79AE3.9010302@syncopated.net>
Date: Sat, 7 Jul 2012 21:51:22 -0500
Message-ID: 
 <CAAam9sty061PkzMu7pvPGiGsu1EoczU+ro+982YznG-1Rx+qkQ@mail.gmail.com>
Subject: Re: node vs node latency
From: Tyler Hobbs <tyler@datastax.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=e89a8f22c34f8f288b04c448921f

--e89a8f22c34f8f288b04c448921f
Content-Type: text/plain; charset=ISO-8859-1

Those latencies look like the difference between a couple of disk seeks and
reading something that's already in the os cache.

The dynamic snitch will favor nodes with lower latencies.  Once a node has
served enough reads, it might not have to hit disk very often, which
produces lower latencies.  So, if you have a hot dataset that fits into
memory, the dynamic snitch starts a positive feedback loop where most reads
will be served from one replica.

I'm guessing the node with the low latencies is serving most of your reads.
You can look at how quickly the total read count is increasing for each of
the replicas to confirm this.  It's not easy to do with only nodetool
cfstats, but something like OpsCenter would help.

On Fri, Jul 6, 2012 at 9:11 PM, Deno Vichas <deno@syncopated.net> wrote:

> all,
>
> what would explain a huge different (12ms vs 0.1ms) in read latency from
> node to node.  i've got a 4 node cluster w/ replication factor of 3 using
> hector.  i'm seeing these numbers with nodetool cfstats.
>
>
> thx,
> deno
>


-- 
Tyler Hobbs
DataStax <http://datastax.com/>

--e89a8f22c34f8f288b04c448921f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Those latencies look like the difference between a couple of disk seeks and=
 reading something that&#39;s already in the os cache.<br><br>The dynamic s=
nitch will favor nodes with lower latencies.=A0 Once a node has served enou=
gh reads, it might not have to hit disk very often, which produces lower la=
tencies.=A0 So, if you have a hot dataset that fits into memory, the dynami=
c snitch starts a positive feedback loop where most reads will be served fr=
om one replica.<br>
<br>I&#39;m guessing the node with the low latencies is serving most of you=
r reads. You can look at how quickly the total read count is increasing for=
 each of the replicas to confirm this.=A0 It&#39;s not easy to do with only=
 nodetool cfstats, but something like OpsCenter would help.<br>
<br><div class=3D"gmail_quote">On Fri, Jul 6, 2012 at 9:11 PM, Deno Vichas =
<span dir=3D"ltr">&lt;<a href=3D"mailto:deno@syncopated.net" target=3D"_bla=
nk">deno@syncopated.net</a>&gt;</span> wrote:<br><blockquote class=3D"gmail=
_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:=
1ex">
all,<br>
<br>
what would explain a huge different (12ms vs 0.1ms) in read latency from no=
de to node. =A0i&#39;ve got a 4 node cluster w/ replication factor of 3 usi=
ng hector. =A0i&#39;m seeing these numbers with nodetool cfstats.<br>
<br>
<br>
thx,<br>
deno<br>
</blockquote></div><br><br clear=3D"all"><br>-- <br><font color=3D"#888888"=
>Tyler Hobbs<span></span><br>
<a href=3D"http://datastax.com/" target=3D"_blank">DataStax</a><br></font><=
br>

--e89a8f22c34f8f288b04c448921f--