cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Caleb Rackliffe <ca...@steelhouse.com>
Subject Re: Lots and Lots of CompactionReducer Threads
Date Sun, 08 Jan 2012 22:48:59 GMT
With the exception of a few little warnings on start-up about the Memtable live ratio, there
is nothing at WARN or above in the logs.  Just before the JVM terminates, there are about
10,000 threads in Reducer executor pools that look like this in JConsole …


Name: CompactionReducer:1
State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
Total blocked: 0  Total waited: 1

Stack trace:
 sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)


The results from tpstats don't look too interesting…

Pool Name                    Active   Pending      Completed   Blocked  All time blocked
ReadStage                         0         0        3455159         0                 0
RequestResponseStage              0         0       10133276         0                 0
MutationStage                     0         0        5898833         0                 0
ReadRepairStage                   0         0        2078449         0                 0
ReplicateOnWriteStage             0         0              0         0                 0
GossipStage                       0         0         236388         0                 0
AntiEntropyStage                  0         0              0         0                 0
MigrationStage                    0         0              0         0                 0
MemtablePostFlusher               0         0            231         0                 0
StreamStage                       0         0              0         0                 0
FlushWriter                       0         0            231         0                 0
MiscStage                         0         0              0         0                 0
InternalResponseStage             0         0              0         0                 0
HintedHandoff                     0         0             35         0                 0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR                  0
BINARY                       0
READ                         0
MUTATION                     0
REQUEST_RESPONSE             0

The results from info seem unremarkable as well…

Token            : 153127065000000000000000000000000000000
Gossip active    : true
Load             : 5.6 GB
Generation No    : 1325995515
Uptime (seconds) : 67199
Heap Memory (MB) : 970.32 / 1968.00
Data Center      : datacenter1
Rack             : rack1
Exceptions       : 0

I'm using LeveledCompactionStrategy with no throttling, and I'm not changing the default on
the number of concurrent compactors.

What is interesting to me here is that Cassandra creates an executor for every single compaction
in ParallelCompactionIterable.  Why couldn't we just create a pool with Runtime.availableProcessors()
Threads and be done with it?

Let me know if I left any info out.

Thanks!

Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com


From: aaron morton <aaron@thelastpickle.com<mailto:aaron@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Sun, 8 Jan 2012 16:51:50 -0500
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Lots and Lots of CompactionReducer Threads

How many threads ? Any errors in the server logs ?

What does noodtool tpstats and nodetool compactionstats say ?

Did you change compaction_strategy for the CF's ?

By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors
in cassandra.yaml

Have you set the JVM heap settings ? What does nodetool info show ?

Hope that helps.

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote:

Hi Everybody,

JConsole tells me I've got CompactionReducer threads stacking up, consuming memory, and never
going away.  Eventually, my Java process fails because it can't allocate any more native threads.
 Here's my setup…

Cassandra 1.0.5 on CentOS 6.0
4 GB of RAM
50 GB SSD HD
Memtable flush threshold = 128 MB
compaction throughput limit = 16 MB/sec
Multithreaded compaction = true

It may very well be that I'm doing something strange here, but it seems like those compaction
threads should go away eventually.  I'm hoping the combination of a low Memtable flush threshold,
low compaction T/P limit, and heavy write load doesn't mean those threads are hanging around
because they're actually not done doing their compaction tasks.

Thanks,

Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<mailto:caleb@steelhouse.com>



Mime
View raw message