incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haithem Jarraya <haithem.jarr...@struq.com>
Subject Thrift threads proliferation
Date Fri, 14 Jun 2013 12:01:28 GMT
Hi All,

We are facing a very strange issue in our C* ring. We are using C* v1.2.4,
7 Nodes in DC1, 3 Nodes in DC2 and 3 Nodes in DC3.
We have been testing read/write performances in DC1, by having different
disks configurations.
For instance we have node1-DC1 use JBOD and node2-DC1 is using RAID-0
configuration.
Over the last week everything seems to be running fine until yesterday when
node2-DC1 (RAID-0) config stop responding to client requests and timing out
queries.
JMX console showed up to 25k Thrift threads running, no pending compaction
running, a lot of pending reads and that's it, CPU is averaging at 10% heap
usage is about 4GB of the 8GB available.
Node2-DC1 become unresponsive but still other node were trying to query it
and it was not flagged as dead or unresponsive from Gossip messages,
wondering if it's a bug.

Log file shows after stopping thrift from node2-DC1:

 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,433 StatusLogger.java (line
53) Pool Name                    Active   Pending   Blocked
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,564 StatusLogger.java (line
68) ReadStage                        30       959         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,564 StatusLogger.java (line
68) RequestResponseStage              0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,565 StatusLogger.java (line
68) ReadRepairStage                   0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,565 StatusLogger.java (line
68) MutationStage                     0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,566 StatusLogger.java (line
68) ReplicateOnWriteStage             0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,566 StatusLogger.java (line
68) GossipStage                       0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,567 StatusLogger.java (line
68) AntiEntropyStage                  0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,567 StatusLogger.java (line
68) MigrationStage                    0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,568 StatusLogger.java (line
68) MemtablePostFlusher               0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,568 StatusLogger.java (line
68) FlushWriter                       0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,569 StatusLogger.java (line
68) MiscStage                         0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,569 StatusLogger.java (line
68) commitlog_archiver                0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,570 StatusLogger.java (line
68) InternalResponseStage             0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,570 StatusLogger.java (line
68) HintedHandoff                     0         0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,572 StatusLogger.java (line
73) CompactionManager                 0         0
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,573 StatusLogger.java (line
85) MessagingService                n/a      0,42
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,574 StatusLogger.java (line
95) Cache Type                     Size                 Capacity
    KeysToSave
Provider
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,574 StatusLogger.java (line
96) KeyCache                  602369792               1048576000
           all
 INFO [ScheduledTasks:1] 2013-06-14 09:29:37,574 StatusLogger.java (line
102) RowCache                          0                        0
           all


 Any hint to track down this error would be useful,


Many Thanks,

Haithem

Mime
View raw message