incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Katkov <ikat...@gmail.com>
Subject Re: Storage proxy write latency is too high
Date Mon, 05 Oct 2009 19:17:35 GMT
measured via JMX console i.e. does not include client-cassandra-client
latency

20 client threads 176975b value StorageProxy.WriteLatency ~660ms
10 client threads 176975b value StorageProxy.WriteLatency ~350ms
05 client threads 176975b value StorageProxy.WriteLatency ~156ms

20 client threads 4978b value StorageProxy.WriteLatency ~32ms
10 client threads 4978b value StorageProxy.WriteLatency ~32ms
05 client threads 4978b value StorageProxy.WriteLatency ~31ms

20 client threads 1000b value StorageProxy.WriteLatency ~30ms
10 client threads 1000b value StorageProxy.WriteLatency ~30ms
05 client threads 1000b value StorageProxy.WriteLatency ~29ms

On Mon, Oct 5, 2009 at 2:52 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> so for a 200KB value you are seeing >200ms of latency in
> MessagingService?  How much for a 1KB value?
>
> -Jonathan
>
> On Mon, Oct 5, 2009 at 1:46 PM, Igor Katkov <ikatkov@gmail.com> wrote:
> > In case anyone is following this, here is an update:
> >
> > I was able to narrow it down to Cassandra-Cassandra link. Storage proxy
> > latency depends on size of the key. The larger amount of data (per key)
> is
> > transfered the larger latency is. No surprise here.
> > Client connects to a demons "A" and sends key-value, "A" accept thrift
> > message, de-serialize it to an object, sees that key belongs to demons
> "B",
> > serialize it to bytes once again (internal format now) and invoke
> > MessagingService, which in turn writes to a socket. As soon as "B"
> delivers
> > write-acknowledgment over a different connection, the client call is let
> go.
> > Cassandra's MessagingService utilizes java nio to connect to other
> cassandra
> > daemons, all connections are uni-directionals. So in theory it should be
> > very fast. But it's not.
> >
> > What does look suspicious is certain network usage cap, only ~4% of the
> > 1Gbps link is used regardless of "value" size. With smaller value I get a
> > better throughput, with larger (200Kb) - worse.
> >
> > As a temp workaround I see that client might be held responsible to
> > identifying what cassandra instance it should send a key to. On 200kb
> value
> > it's ~10 times faster.
> >
> >
> > On Thu, Oct 1, 2009 at 6:51 PM, Igor Katkov <ikatkov@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> I have the following puzzle:
> >> Storage proxy write latency ~235ms
> >> CF write latency <1 ms
> >>
> >> I have 3 nodes in the cluster, Cassandra v.0.4. Tokens evenly
> distributed.
> >> The client connects to a node and inserts a key with
> ConsistencyLevel.ONE
> >> If it happen to be a local write operation is fast, same speed as in one
> >> node setup. JMX shows write latency <1 ms
> >> If it happens to be a remote insert StorageProxy sends it to a proper
> >> node. This operation is slow. JMX shows write latency ~ 235ms.
> >> In the same time, on remote node JMX shows same <1ms write latency. So
> >> it's not remote node being sluggish, it's something else.
> >> There are no pending tasks on remote node - JMX counters are always
> zero,
> >> network is 1Gb, idle. So I can't blame it.
> >>
> >>
> >> I profiled Cassandra server in JProfiler, could not find a thing. All
> this
> >> extra time is spent inside QuorumResponseHandler waiting for the
> condition
> >> to signal. Which should happen as soon as response is received.
> >>
> >> There is one pooled TCP connection open to remote host. Hardly a
> >> bottleneck, ThreadPoolExecutors looks OK.
> >>
> >> Any ideas why write latency it is so high?
> >
> >
>

Mime
View raw message