incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron <aa...@thelastpickle.com>
Subject Re: stalled streaming
Date Tue, 15 Jun 2010 23:06:06 GMT
Thanks, will move to 0.6.2. 

Aaron

On Tue, 15 Jun 2010 15:55:46 -0700, Benjamin Black <b@b3k.us> wrote:
> Known bug, fixed in latest 0.6 release.
> 
> On Tue, Jun 15, 2010 at 3:29 PM, aaron <aaron@thelastpickle.com> wrote:
>> hello,
>>
>> I have a 4 node cassandra cluster with 0.6.1 installed. We've been
>> running
>> a mixed read / write workload test how it works in our environment, we
>> run
>> about 4M bath mutations and 40M get_range_slice requests over 6 to 8
>> hours
>> that load about 10 to 15 GB of data.
>>
>> Yesterday while there was no activity I noticed 2 nodes sitting at 200%
>> CPU on 8 Core machine. Thought nothing of it. Checked again this
morning
>> and they are still sitting at that level of activity with no requests
>> going
>> into them. Checking the streams using node tool I see node 3 is
streaming
>> to node 0 and 2, and appears to have stalled. The information in the
JMX
>> console for streams matches the info below.
>>
>> I cannot see any errors in the logs.
>>
>> This is just a test system, so am happy to bounce the JVM's. Before I
do
>> is there anything else I should be looking for to understand why this
>> happened?
>>
>> Also, sorry for the previous empty email.
>>
>> Node 0
>> Mode: Normal
>>  Nothing streaming to /192.168.34.27
>>  Nothing streaming to /192.168.34.28
>>  Nothing streaming to /192.168.34.29
>> Streaming from: /192.168.34.29
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db
>> 0/22765
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db
>> 0/10750717
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Index.db
>> 0/58
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Filter.db
>> 0/325
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Data.db
>> 0/695
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Index.db
>> 0/58
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Filter.db
>> 0/325
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Data.db
>> 0/695
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Index.db
>> 0/587164
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Filter.db
>> 0/22765
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Data.db
>> 0/5159652
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-124-Data.db
>> 22765/4966927
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Index.db
>> 22765/1223053
>>
>> Node 1
>> Mode: Normal
>>  Nothing streaming to /192.168.34.26
>>  Nothing streaming to /192.168.34.28
>>  Nothing streaming to /192.168.34.29
>> Not receiving any streams.
>>
>> Node 2
>> Mode: Normal
>>  Nothing streaming to /192.168.34.26
>>  Nothing streaming to /192.168.34.27
>>  Nothing streaming to /192.168.34.29
>> Streaming from: /192.168.34.29
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db
>> 0/22765
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db
>> 0/2161660
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Index.db
>> 0/787524
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Filter.db
>> 0/22765
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Data.db
>> 0/6917064
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Index.db
>> 0/58
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Filter.db
>> 0/565
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Data.db
>> 0/695
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Index.db
>> 0/581779
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Filter.db
>> 0/22765
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Data.db
>> 0/5111887
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Data.db
>> 361367/3173057
>>   junkbox.mycompany:
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Index.db
>> 695/361367
>>
>> Node 3
>> ode: Normal
>> Streaming to: /192.168.34.26
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Filter.db
>> 22765/22765
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Data.db
>> 0/4966927
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Index.db
>> 0/58
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Filter.db
>> 0/325
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Data.db
>> 0/695
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Index.db
>> 0/1223053
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Filter.db
>> 0/22765
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Data.db
>> 0/10750717
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Index.db
>> 0/58
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Filter.db
>> 0/325
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Data.db
>> 0/695
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Index.db
>> 0/58
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Filter.db
>> 0/325
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Data.db
>> 0/695
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Index.db
>> 0/587164
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Filter.db
>> 0/22765
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Data.db
>> 0/5159652
>>  Nothing streaming to /192.168.34.27
>> Streaming to: /192.168.34.28
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Index.db
>> 361367/361367
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Filter.db
>> 0/22765
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Data.db
>> 0/3173057
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Index.db
>> 0/58
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Filter.db
>> 0/325
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Data.db
>> 0/695
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Index.db
>> 0/246014
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Filter.db
>> 0/22765
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Data.db
>> 0/2161660
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Index.db
>> 0/787524
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Filter.db
>> 0/22765
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Data.db
>> 0/6917064
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Index.db
>> 0/58
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Filter.db
>> 0/565
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Data.db
>> 0/695
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Index.db
>> 0/581779
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Filter.db
>> 0/22765
>>
>>
/local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Data.db
>> 0/5111887
>> Not receiving any streams.
>>
>>
>>
>>

Mime
View raw message