incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe <watche...@gmail.com>
Subject Re: CPU bound workload
Date Mon, 12 Dec 2011 09:29:27 GMT
>
> Ah, I keep always assuming random partitions since it is a very common
> case (just to be sure: unless you specifically want the ordering
> despite the downsides, you generally want to default to the random
> partitioner).
>
Yes, I'm working on geographical data so everything is keyed by a
derivation of the z-order curve. Reads are such that they read linear
ranges of the values. So although there's data in a lot of places, writes &
reads happen more often in only a subset hence my need to rebalance not by
storage load but by query load. I'm going to have to think about how to do
that..


> > When I get a log like these, there always is a "cluster-freeze" during
> the
> > preceding minute. By "cluster-freeze", I mean that a couple of nodes go
> to
> > 0% utilization (no cpu, no system, no io)
> An hypothesis here is that your workload is causing problem for a node
> (for example, sudden spikes in memory allocation causing full GC
> fallbacks that take time), and both the readers and the writers get
> "stuck" on requests to those nodes (once a sufficient number of
> requests happen to be destined to those). The result would be that all
> other nodes are no longer seeing traffic because the clients aren't
> making progress.
>
I did open 10 windows on my screen in order to view iostat & vmstat in
parallel on all nodes. It's hard to be definitive but I did see moments
where the cluster was "freezing" and the CPU was at 100% for a couple
seconds on one of the nodes. It didn't happen everytime but that may be
because my processes were backing off while failing-over to another node.

So could sending a batch of 90 counter mutations produce such a spike ? It
looks a little small to me but, yes, maxing the batches to 30/40 elements
has eliminated the problem.I looked at the system.log on the node on which
this happened and I only see a ParNew collection taking place at that time
and heap usage is low.
INFO [ScheduledTasks:1] 2011-12-11 15:48:14,641 GCInspector.java (line 122)
GC for ParNew: 334 ms for 1 collections, 4729419512 used; max is 16838033408



> I would first eliminate or confirm any GC hypothesis by running all
> nodes with -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -XX:+PrintGCDateStamps.
>
Is full GC not being logged through GCInspector with the defaults ?


> If you can see this happen sufficiently often to
> manually/interactively "wait for it", I suggest something as simple as
> fireing up an top + iostat for each host and have them on the screen
> at the same time, and look for what happens when you see this again.
> If the problem is fallback to full GC for example, the affected nodes
> should be churning 100% CPU (one core) for several seconds (assuming a
> large heap). If there is a sudden burst of disk I/O that is causing a
> hiccup (e.g. dirty buffer flushing by linux) this should be visibly
> correlated with 'iostat -x -k 1'.
>
some CPU correlation in some cases (on one node)
no iostat correlation ever


Thanks
Philippe

Mime
View raw message