cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Mischo <jmis...@quagility.com>
Subject Re: Storage proxy write latency is too high
Date Tue, 06 Oct 2009 22:31:14 GMT
Back in the day, I was involved in performance testing JVMs, Solaris  
on Intel, and SWS at SunSoft, and our lab actually ran not just our  
own numbers, but competitors' numbers as well, for comparison.  One  
thing we discovered, when analyzing some performance issues we'd seen  
with Windows clients, was that Windows' network stacks don't behave as  
you'd expect.  They very regularly transmitted packets out of sequence.

I'm not sure if this is still the case, as this was the late 90's, but  
we discovered this when we were working on SPECweb numbers (we were a  
SPEC lab) for SWS and, in looking at the JVM and other system settings  
to understand why we were seeing unexpectedly large TCP buffers.  When  
we started sniffing packets directly from the Windows clients, we  
discovered the packets were being emitted out of sequence, which was  
causing the server to require larger per-connection buffers and was  
pushing TCP window size boundaries.

Now, Cassandra uses UDP primarily, but we never tested to find out  
whether this was a TCP stack issue, IP stack issue, or Ethernet stack  
issue, so it may be a similar case.  Since reassembly still has to  
happen, if the packets are being transmitted out of order, it makes  
sense that your write latency would be significantly higher.

I'd be interested to see the results if you dig deeper into this.

-Jon

On Oct 6, 2009, at 11:14 AM, Igor Katkov wrote:

> I think I finally found what. It's implementation of Java NIOon  
> Windows (JVM 1.6.0.16, 64b on Windows 2003)
> The very same code, same network but CentOS linux gives almost 4x  
> performance. (in Cassandra@linux -> Cassandra@Windows setup)
> I don't have another linux box to test (Cassandra@linux ->  
> Cassandra@linux) performance, but expect it to be even better.
>
> A lesson learnt: don't use windows.
>
> P.S.
> Here at Viigo we also learnt the hard way that  async IO is also  
> broken in .NET (C#). Now I start to wonder if there is some  
> fundamental flaw in async IO on windows...
>
> On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jbellis@gmail.com>  
> wrote:
> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ikatkov@gmail.com> wrote:
> > measured via JMX console i.e. does not include client-cassandra- 
> client
> > latency
> >
> > 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
> > 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
> > 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>
> this is going up basically linearly with amount of (data x clients),
> so clearly something is getting saturated.
>


Mime
View raw message