incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <>
Subject Re: Storage proxy write latency is too high
Date Tue, 06 Oct 2009 23:26:02 GMT
Just to clarify, Cassandra only uses UDP for gossip -- passing data
between nodes is done over TCP.


On Tue, Oct 6, 2009 at 5:31 PM, Jonathan Mischo <> wrote:
> Back in the day, I was involved in performance testing JVMs, Solaris on
> Intel, and SWS at SunSoft, and our lab actually ran not just our own
> numbers, but competitors' numbers as well, for comparison.  One thing we
> discovered, when analyzing some performance issues we'd seen with Windows
> clients, was that Windows' network stacks don't behave as you'd expect.
>  They very regularly transmitted packets out of sequence.
> I'm not sure if this is still the case, as this was the late 90's, but we
> discovered this when we were working on SPECweb numbers (we were a SPEC lab)
> for SWS and, in looking at the JVM and other system settings to understand
> why we were seeing unexpectedly large TCP buffers.  When we started sniffing
> packets directly from the Windows clients, we discovered the packets were
> being emitted out of sequence, which was causing the server to require
> larger per-connection buffers and was pushing TCP window size boundaries.
> Now, Cassandra uses UDP primarily, but we never tested to find out whether
> this was a TCP stack issue, IP stack issue, or Ethernet stack issue, so it
> may be a similar case.  Since reassembly still has to happen, if the packets
> are being transmitted out of order, it makes sense that your write latency
> would be significantly higher.
> I'd be interested to see the results if you dig deeper into this.
> -Jon
> On Oct 6, 2009, at 11:14 AM, Igor Katkov wrote:
> I think I finally found what. It's implementation of Java NIOon Windows (JVM
>, 64b on Windows 2003)
> The very same code, same network but CentOS linux gives almost 4x
> performance. (in Cassandra@linux -> Cassandra@Windows setup)
> I don't have another linux box to test (Cassandra@linux -> Cassandra@linux)
> performance, but expect it to be even better.
> A lesson learnt: don't use windows.
> P.S.
> Here at Viigo we also learnt the hard way that  async IO is also broken in
> .NET (C#). Now I start to wonder if there is some fundamental flaw in async
> IO on windows...
> On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <> wrote:
>> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <> wrote:
>> > measured via JMX console i.e. does not include client-cassandra-client
>> > latency
>> >
>> > 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
>> > 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
>> > 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>> this is going up basically linearly with amount of (data x clients),
>> so clearly something is getting saturated.

View raw message