What Jonathan said.  Also, if you have the ability to switch your profiler between wall mode and CPU mode, I would recommend it to give you a better overall picture of what is going on.  If your profiler can switch between 'sampling' and 'tracing' modes that would also be useful.

I'm not sure if that picture and the profiling description is intended to explain the need for incremental GC, but I don't understand it.  What difference in memory growth are you seeing?  I'm not sure I can explain that based on the GC docs I've read (although I'll be the first to admit I have a relatively limited understanding of how it works)

Michael

On Wed, Aug 19, 2009 at 7:37 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
be careful when profiling blocking io -- I bet that means that "I'm
spending all my time blocking for more data to read since there is
only one call per second."

the internal Cassandra MessagingService uses nonblocking io, but the
Thrift stuff is just your standard thread pool with blocking sockets.

On Wed, Aug 19, 2009 at 5:32 PM, Huming Wu<huming.wu@gmail.com> wrote:
> On Tue, Aug 18, 2009 at 12:02 PM, Michael
> Greene<michael.greene@gmail.com> wrote:
>> According to the HBase guys and confirmed by
>> http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html#icms,
>> on an 8-core machine you are not going to want to enable
>> -XX:+CMSIncrementalMode -- it is for 1 or 2 core machines that are
>> using the CMS GC.  This should affect your latency numbers positively,
>> but I wouldn't see it changing the CPU usage much.
>
> Looks I do need to use the incremental GC though otherwise the memory
> growth is very fast... On the get_slice CPU, I tried the profiling (on
> single node and low load 1 get_slice call per second) and it looks
> org.apache.thrift.transport.TIOStreamTransport.read sucks up all CPU.
> Here is the snapshot
> http://farm3.static.flickr.com/2595/3838562412_8ffb42ea8c_o.png
> (sounds an issue in thrift). Any idea?
>
> Thanks,
> Huming
>