incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Excessive allocation during hinted handoff
Date Thu, 12 May 2011 02:06:37 GMT
I'm assuming the two nodes are the ones receiving the HH after they were down. 

Are there a lot of hints collected while they are down ? you can check the HintedHandOffManager
MBean in JConsole

What does the TPStats look like on the nodes under pressure ? And how many nodes are delivering
hints to the nodes when they restart?

Finally hinted_handoff_throttle_delay_in_ms in conf/cassandra.yaml will let you slow down
the delivery rate if HH is indeed the problem. 

Hope that helps.

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 12 May 2011, at 06:55, Gabriel Tataranu wrote:

> Greetings,
> 
> I'm experiencing some issues with 2 nodes (out of more than 10). Right
> after startup (Listening for thrift clients...) the nodes will create
> objects at high rate using all available CPU cores:
> 
> INFO 18:13:15,350 GC for PS Scavenge: 292 ms, 494902976 reclaimed
> leaving 2024909864 used; max is 6658457600
> INFO 18:13:20,393 GC for PS Scavenge: 252 ms, 478691280 reclaimed
> leaving 2184252600 used; max is 6658457600
> ....
> INFO 18:15:23,909 GC for PS Scavenge: 283 ms, 452943472 reclaimed
> leaving 5523891120 used; max is 6658457600
> INFO 18:15:24,912 GC for PS Scavenge: 273 ms, 466157568 reclaimed
> leaving 5594606128 used; max is 6658457600
> 
> This will eventually trigger old-gen GC and then the process repeats
> until hinted handoff finishes.
> 
> The build version was updated from 0.7.2 to 0.7.5 but the behavior was
> exactly the same.
> 
> Thank you.
> 


Mime
View raw message