On 05/18/2011 09:28 AM, Cassa L wrote:
Hi,
  I have 9 node cluster with RF-3 and using Cassandra0.70/Hector26. Recently we are seeing lot of "UnavailableException" at the client side. Whenever this happens, I found following pattern in Cassandra node's log file at that given time,

 INFO [ScheduledTasks:1] 2011-05-13 02:59:55,365 Gossiper.java (line 195) InetAddress /**.**.***.54 is now dead.
 INFO [ScheduledTasks:1] 2011-05-13 02:59:57,369 Gossiper.java (line 195) InetAddress /**.**.***.59 is now dead.
 INFO [HintedHandoff:1] 2011-05-13 03:00:04,706 HintedHandOffManager.java (line 192) Started hinted handoff for endpoint /**.**.***.54
 INFO [GossipStage:1] 2011-05-13 03:00:04,706 Gossiper.java (line 569) InetAddress /**.**.***.54 is now UP
 INFO [HintedHandoff:1] 2011-05-13 03:00:04,706 HintedHandOffManager.java (line 248) Finished hinted handoff of 0 rows to endpoint /**.**.***.54
 INFO [HintedHandoff:1] 2011-05-13 03:00:20,601 HintedHandOffManager.java (line 192) Started hinted handoff for endpoint /**.**.***.59
 INFO [GossipStage:1] 2011-05-13 03:00:20,601 Gossiper.java (line 569) InetAddress /**.**.***.59 is now UP
 
The exception occurred at "2011-05-13 03:00:00,664". I am wondering what why this dead/up pattern is occurring at Gossip.

Thanks in advance,
Cassa L.

I am not sure do check system Load when you see this happening.If load is to High then it means CPU was not able to give time to Network I/O.
-- 
S.Ali Ahsan