incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron <aa...@thelastpickle.com>
Subject stalled streaming
Date Tue, 15 Jun 2010 22:29:30 GMT
hello, 

I have a 4 node cassandra cluster with 0.6.1 installed. We've been running
a mixed read / write workload test how it works in our environment, we run
about 4M bath mutations and 40M get_range_slice requests over 6 to 8 hours
that load about 10 to 15 GB of data. 

Yesterday while there was no activity I noticed 2 nodes sitting at 200%
CPU on 8 Core machine. Thought nothing of it. Checked again this morning
and they are still sitting at that level of activity with no requests going
into them. Checking the streams using node tool I see node 3 is streaming
to node 0 and 2, and appears to have stalled. The information in the JMX
console for streams matches the info below.

I cannot see any errors in the logs. 

This is just a test system, so am happy to bounce the JVM's. Before I do
is there anything else I should be looking for to understand why this
happened?

Also, sorry for the previous empty email. 

Node 0
Mode: Normal
 Nothing streaming to /192.168.34.27
 Nothing streaming to /192.168.34.28
 Nothing streaming to /192.168.34.29
Streaming from: /192.168.34.29
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db
0/22765
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db
0/10750717
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Index.db
0/58
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Filter.db
0/325
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Data.db
0/695
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Index.db
0/58
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Filter.db
0/325
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Data.db
0/695
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Index.db
0/587164
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Filter.db
0/22765
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Data.db
0/5159652
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-124-Data.db
22765/4966927
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Index.db
22765/1223053

Node 1
Mode: Normal
 Nothing streaming to /192.168.34.26
 Nothing streaming to /192.168.34.28
 Nothing streaming to /192.168.34.29
Not receiving any streams.

Node 2
Mode: Normal
 Nothing streaming to /192.168.34.26
 Nothing streaming to /192.168.34.27
 Nothing streaming to /192.168.34.29
Streaming from: /192.168.34.29
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db
0/22765
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db
0/2161660
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Index.db
0/787524
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Filter.db
0/22765
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Data.db
0/6917064
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Index.db
0/58
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Filter.db
0/565
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Data.db
0/695
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Index.db
0/581779
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Filter.db
0/22765
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Data.db
0/5111887
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Data.db
361367/3173057
   junkbox.mycompany:
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Index.db
695/361367

Node 3
ode: Normal
Streaming to: /192.168.34.26
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Filter.db
22765/22765
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Data.db
0/4966927
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Index.db
0/58
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Filter.db
0/325
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Data.db
0/695
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Index.db
0/1223053
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Filter.db
0/22765
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Data.db
0/10750717
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Index.db
0/58
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Filter.db
0/325
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Data.db
0/695
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Index.db
0/58
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Filter.db
0/325
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Data.db
0/695
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Index.db
0/587164
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Filter.db
0/22765
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Data.db
0/5159652
 Nothing streaming to /192.168.34.27
Streaming to: /192.168.34.28
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Index.db
361367/361367
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Filter.db
0/22765
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Data.db
0/3173057
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Index.db
0/58
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Filter.db
0/325
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Data.db
0/695
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Index.db
0/246014
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Filter.db
0/22765
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Data.db
0/2161660
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Index.db
0/787524
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Filter.db
0/22765
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Data.db
0/6917064
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Index.db
0/58
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Filter.db
0/565
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Data.db
0/695
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Index.db
0/581779
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Filter.db
0/22765
  
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Data.db
0/5111887
Not receiving any streams.


 

Mime
View raw message