incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Mischo <jmis...@quagility.com>
Subject Re: Storage proxy write latency is too high
Date Wed, 07 Oct 2009 21:52:21 GMT
My bad...I guess that makes it even more important to find out if this  
is the problem on Windows? :)

-Jon

On Oct 6, 2009, at 6:26 PM, Jonathan Ellis wrote:

> Just to clarify, Cassandra only uses UDP for gossip -- passing data
> between nodes is done over TCP.
>
> -Jonathan
>
> On Tue, Oct 6, 2009 at 5:31 PM, Jonathan Mischo  
> <jmischo@quagility.com> wrote:
>> Back in the day, I was involved in performance testing JVMs,  
>> Solaris on
>> Intel, and SWS at SunSoft, and our lab actually ran not just our own
>> numbers, but competitors' numbers as well, for comparison.  One  
>> thing we
>> discovered, when analyzing some performance issues we'd seen with  
>> Windows
>> clients, was that Windows' network stacks don't behave as you'd  
>> expect.
>>  They very regularly transmitted packets out of sequence.
>> I'm not sure if this is still the case, as this was the late 90's,  
>> but we
>> discovered this when we were working on SPECweb numbers (we were a  
>> SPEC lab)
>> for SWS and, in looking at the JVM and other system settings to  
>> understand
>> why we were seeing unexpectedly large TCP buffers.  When we started  
>> sniffing
>> packets directly from the Windows clients, we discovered the  
>> packets were
>> being emitted out of sequence, which was causing the server to  
>> require
>> larger per-connection buffers and was pushing TCP window size  
>> boundaries.
>> Now, Cassandra uses UDP primarily, but we never tested to find out  
>> whether
>> this was a TCP stack issue, IP stack issue, or Ethernet stack  
>> issue, so it
>> may be a similar case.  Since reassembly still has to happen, if  
>> the packets
>> are being transmitted out of order, it makes sense that your write  
>> latency
>> would be significantly higher.
>> I'd be interested to see the results if you dig deeper into this.
>> -Jon
>>
>> On Oct 6, 2009, at 11:14 AM, Igor Katkov wrote:
>>
>> I think I finally found what. It's implementation of Java NIOon  
>> Windows (JVM
>> 1.6.0.16, 64b on Windows 2003)
>> The very same code, same network but CentOS linux gives almost 4x
>> performance. (in Cassandra@linux -> Cassandra@Windows setup)
>> I don't have another linux box to test (Cassandra@linux ->  
>> Cassandra@linux)
>> performance, but expect it to be even better.
>>
>> A lesson learnt: don't use windows.
>>
>> P.S.
>> Here at Viigo we also learnt the hard way that  async IO is also  
>> broken in
>> .NET (C#). Now I start to wonder if there is some fundamental flaw  
>> in async
>> IO on windows...
>>
>> On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jbellis@gmail.com>  
>> wrote:
>>>
>>> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ikatkov@gmail.com>  
>>> wrote:
>>>> measured via JMX console i.e. does not include client-cassandra- 
>>>> client
>>>> latency
>>>>
>>>> 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
>>>> 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
>>>> 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>>>
>>> this is going up basically linearly with amount of (data x clients),
>>> so clearly something is getting saturated.
>>
>>
>>
>


Mime
View raw message