cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Lots and Lots of CompactionReducer Threads
Date Mon, 09 Jan 2012 09:52:55 GMT
Thanks, good work. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 9/01/2012, at 1:25 PM, Caleb Rackliffe wrote:

> After some searching, I think I may have found something in the code itself, and so I've
filed a big report - https://issues.apache.org/jira/browse/CASSANDRA-3711
> 
> Caleb Rackliffe | Software Developer	
> M 949.981.0159 | caleb@steelhouse.com
> 
> 
> From: Caleb Rackliffe <caleb@steelhouse.com>
> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Date: Sun, 8 Jan 2012 17:48:59 -0500
> To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Cc: "aaron@thelastpickle.com" <aaron@thelastpickle.com>
> Subject: Re: Lots and Lots of CompactionReducer Threads
> 
> With the exception of a few little warnings on start-up about the Memtable live ratio,
there is nothing at WARN or above in the logs.  Just before the JVM terminates, there are
about 10,000 threads in Reducer executor pools that look like this in JConsole …
> 
> 
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0  Total waited: 1
> 
> Stack trace: 
>  sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
> 
> 
> The results from tpstats don't look too interesting…
> 
> Pool Name                    Active   Pending      Completed   Blocked  All time blocked
> ReadStage                         0         0        3455159         0              
  0
> RequestResponseStage              0         0       10133276         0              
  0
> MutationStage                     0         0        5898833         0              
  0
> ReadRepairStage                   0         0        2078449         0              
  0
> ReplicateOnWriteStage             0         0              0         0              
  0
> GossipStage                       0         0         236388         0              
  0
> AntiEntropyStage                  0         0              0         0              
  0
> MigrationStage                    0         0              0         0              
  0
> MemtablePostFlusher               0         0            231         0              
  0
> StreamStage                       0         0              0         0              
  0
> FlushWriter                       0         0            231         0              
  0
> MiscStage                         0         0              0         0              
  0
> InternalResponseStage             0         0              0         0              
  0
> HintedHandoff                     0         0             35         0              
  0
> 
> Message type           Dropped
> RANGE_SLICE                  0
> READ_REPAIR                  0
> BINARY                       0
> READ                         0
> MUTATION                     0
> REQUEST_RESPONSE             0
> 
> The results from info seem unremarkable as well…
> 
> Token            : 153127065000000000000000000000000000000
> Gossip active    : true
> Load             : 5.6 GB
> Generation No    : 1325995515
> Uptime (seconds) : 67199
> Heap Memory (MB) : 970.32 / 1968.00
> Data Center      : datacenter1
> Rack             : rack1
> Exceptions       : 0
> 
> I'm using LeveledCompactionStrategy with no throttling, and I'm not changing the default
on the number of concurrent compactors.
> 
> What is interesting to me here is that Cassandra creates an executor for every single
compaction in ParallelCompactionIterable.  Why couldn't we just create a pool with Runtime.availableProcessors()
Threads and be done with it?
> 
> Let me know if I left any info out.
> 
> Thanks!
> 
> Caleb Rackliffe | Software Developer	
> M 949.981.0159 | caleb@steelhouse.com
> 
> 
> From: aaron morton <aaron@thelastpickle.com>
> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Date: Sun, 8 Jan 2012 16:51:50 -0500
> To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Subject: Re: Lots and Lots of CompactionReducer Threads
> 
> How many threads ? Any errors in the server logs ? 
> 
> What does noodtool tpstats and nodetool compactionstats say ? 
> 
> Did you change compaction_strategy for the CF's ? 
> 
> By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors
in cassandra.yaml
> 
> Have you set the JVM heap settings ? What does nodetool info show ? 
> 
> Hope that helps. 
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote:
> 
>> Hi Everybody,
>> 
>> JConsole tells me I've got CompactionReducer threads stacking up, consuming memory,
and never going away.  Eventually, my Java process fails because it can't allocate any more
native threads.  Here's my setup…
>> 
>> Cassandra 1.0.5 on CentOS 6.0
>> 4 GB of RAM
>> 50 GB SSD HD
>> Memtable flush threshold = 128 MB
>> compaction throughput limit = 16 MB/sec
>> Multithreaded compaction = true
>> 
>> It may very well be that I'm doing something strange here, but it seems like those
compaction threads should go away eventually.  I'm hoping the combination of a low Memtable
flush threshold, low compaction T/P limit, and heavy write load doesn't mean those threads
are hanging around because they're actually not done doing their compaction tasks.
>> 
>> Thanks,
>> 
>> Caleb Rackliffe | Software Developer	
>> M 949.981.0159 | caleb@steelhouse.com
>> 
> 


Mime
View raw message