Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of rohit2412@gmail.com designates
 209.85.214.172 as permitted sender)
MIME-Version: 1.0
Date: Fri, 13 Jul 2012 09:18:20 +0530
Message-ID: 
 <CAGXkHaz8gWHskW6p_FAt-BVAut7Vso1GCCh7NSTLv+rrr9Dx6w@mail.gmail.com>
Subject: High RecentWriteLatencyMicro
From: rohit bhatia <rohit2412@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi

As I understand that writes in cassandra are directly pushed to memory
and using counters with CL.ONE shouldn't take the read latency for
counters in account. So Writes for incrementing counters with CL.ONE
should basically be really fast.

But in my 8 node cluster(16 core/32G ram/cassandra1.0.5/java7 each)
with RF=2, At a traffic of 55k qps = 14k increments per node/7k write
requests per node, the write latency(from jmx) increases to around 7-8
ms from the low traffic value of 0.5ms.  The Nodes aren't even pushed
with absent I/O, lots of free RAM and 30% CPU idle time/OS Load 20.
The write latency by cfstats (supposedly the latency for 1 node to
increment its counter) is a small amount (< 0.05ms).

1) Is the whole of 7-8ms being spent in thrift overheads and
Scheduling delays ? (there is insignificant .1ms ping time between
machines)

2) Do keeping a large number of CF(17 in our case) adversely affect
write performance? (except from the extreme flushing scenario)

3) I see a lot of threads(4,000-10,000) with names like
"pool-2-thread-*" (pointed out as client-connection-threads on the
mailing list before) periodically forming up. but with idle cpu time
and zero pending tasks in tpstats, why do requests keep piling up (GC
stops threads for 100ms every 1-2 seconds, effectively pausing
cassandra 5-10% of its time, but this doesn't seem to be the reason)

Thanks
Rohit