cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Black...@b3k.us>
Subject Re: stalled streaming
Date Tue, 15 Jun 2010 22:55:46 GMT
Known bug, fixed in latest 0.6 release.

On Tue, Jun 15, 2010 at 3:29 PM, aaron <aaron@thelastpickle.com> wrote:
> hello,
>
> I have a 4 node cassandra cluster with 0.6.1 installed. We've been running
> a mixed read / write workload test how it works in our environment, we run
> about 4M bath mutations and 40M get_range_slice requests over 6 to 8 hours
> that load about 10 to 15 GB of data.
>
> Yesterday while there was no activity I noticed 2 nodes sitting at 200%
> CPU on 8 Core machine. Thought nothing of it. Checked again this morning
> and they are still sitting at that level of activity with no requests going
> into them. Checking the streams using node tool I see node 3 is streaming
> to node 0 and 2, and appears to have stalled. The information in the JMX
> console for streams matches the info below.
>
> I cannot see any errors in the logs.
>
> This is just a test system, so am happy to bounce the JVM's. Before I do
> is there anything else I should be looking for to understand why this
> happened?
>
> Also, sorry for the previous empty email.
>
> Node 0
> Mode: Normal
>  Nothing streaming to /192.168.34.27
>  Nothing streaming to /192.168.34.28
>  Nothing streaming to /192.168.34.29
> Streaming from: /192.168.34.29
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db
> 0/22765
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db
> 0/10750717
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Index.db
> 0/58
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Filter.db
> 0/325
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Data.db
> 0/695
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Index.db
> 0/58
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Filter.db
> 0/325
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Data.db
> 0/695
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Index.db
> 0/587164
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Filter.db
> 0/22765
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Data.db
> 0/5159652
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-124-Data.db
> 22765/4966927
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Index.db
> 22765/1223053
>
> Node 1
> Mode: Normal
>  Nothing streaming to /192.168.34.26
>  Nothing streaming to /192.168.34.28
>  Nothing streaming to /192.168.34.29
> Not receiving any streams.
>
> Node 2
> Mode: Normal
>  Nothing streaming to /192.168.34.26
>  Nothing streaming to /192.168.34.27
>  Nothing streaming to /192.168.34.29
> Streaming from: /192.168.34.29
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db
> 0/22765
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db
> 0/2161660
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Index.db
> 0/787524
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Filter.db
> 0/22765
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Data.db
> 0/6917064
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Index.db
> 0/58
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Filter.db
> 0/565
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Data.db
> 0/695
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Index.db
> 0/581779
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Filter.db
> 0/22765
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Data.db
> 0/5111887
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Data.db
> 361367/3173057
>   junkbox.mycompany:
> /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Index.db
> 695/361367
>
> Node 3
> ode: Normal
> Streaming to: /192.168.34.26
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Filter.db
> 22765/22765
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Data.db
> 0/4966927
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Index.db
> 0/58
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Filter.db
> 0/325
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Data.db
> 0/695
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Index.db
> 0/1223053
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Filter.db
> 0/22765
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Data.db
> 0/10750717
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Index.db
> 0/58
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Filter.db
> 0/325
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Data.db
> 0/695
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Index.db
> 0/58
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Filter.db
> 0/325
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Data.db
> 0/695
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Index.db
> 0/587164
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Filter.db
> 0/22765
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Data.db
> 0/5159652
>  Nothing streaming to /192.168.34.27
> Streaming to: /192.168.34.28
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Index.db
> 361367/361367
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Filter.db
> 0/22765
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Data.db
> 0/3173057
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Index.db
> 0/58
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Filter.db
> 0/325
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Data.db
> 0/695
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Index.db
> 0/246014
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Filter.db
> 0/22765
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Data.db
> 0/2161660
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Index.db
> 0/787524
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Filter.db
> 0/22765
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Data.db
> 0/6917064
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Index.db
> 0/58
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Filter.db
> 0/565
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Data.db
> 0/695
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Index.db
> 0/581779
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Filter.db
> 0/22765
>
> /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Data.db
> 0/5111887
> Not receiving any streams.
>
>
>
>

Mime
View raw message