incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <ran...@gmail.com>
Subject Re: nodetool move looks stuck
Date Wed, 26 May 2010 20:12:36 GMT
I don't see any errors in neither the receiver nor sender. The cluster
itself looks healthy (e.g. no random networks disconnections or anything
like that as described in 1019).

I'll restart the moved node, hope it'll continue its more afterwards but it
does look like a bug. Is there something I can do to help debug it?
I'll attach a thread dump here just in case before I restart it



On Wed, May 26, 2010 at 9:54 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> Are there any exceptions in the log like the one in
> https://issues.apache.org/jira/browse/CASSANDRA-1019 ?
>
> If so you'll need to restart the moving node and try again.
>
> On Wed, May 26, 2010 at 3:54 AM, Ran Tavory <rantav@gmail.com> wrote:
> > I ran nodetool move on one of the nodes and it seems stuck for a few
> hours
> > now.
> > I've been able to run it successfully in the past, but this time it looks
> > stuck.
> > Streams shows as if there's work in progress, but the same files have
> been
> > at the same position for a few hours.
> > I've also checked the compaction manager through jmx and it's not
> compacting
> > anything.
> > Anything I can do to continue the move operation? Is this a bug?
> > thanks
> >
> > $ nodetool -h 192.168.254.58 -p 9004 streams
> > Mode: Normal
> >  Nothing streaming to /192.168.252.124
> > Streaming to: /192.168.254.57
> >
> /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Filter.db
> > 0/45587965
> >
> /outbrain/cassandra/data/outbrain_kvdb/stream/KvImpressions-170-Data.db
> > 0/18369406636
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Index.db
> > 0/6163411
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Filter.db
> > 0/85645
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvRatings-142-Data.db
> > 0/37985032
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Index.db
> > 0/77307418
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Filter.db
> > 0/1436005
> >    /outbrain/cassandra/data/outbrain_kvdb/stream/KvAds-108-Data.db
> > 0/704736670
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Index.db
> 0/230280
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Filter.db
> 0/20605
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-40-Data.db
> > 0/202808000
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Index.db
> 0/23912
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Filter.db
> 0/20605
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-41-Data.db
> 0/21434973
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Index.db
> 0/965028
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Filter.db
> 0/20605
> >    /outbrain/cassandra/data/Keyspace1/stream/Standard1-42-Data.db
> > 0/865380205
> > Not receiving any streams.
> > $ nodetool -h 192.168.254.57 -p 9004 streams
> > Mode: Normal
> >  Nothing streaming to /192.168.254.58
> > Streaming from: /192.168.254.58
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvImpressions-tmp-1081-Data.db
> > 0/18369406636
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Index.db
> 0/6163411
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Filter.db
> 0/85645
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvRatings-tmp-167-Data.db
> 0/37985032
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Index.db 0/77307418
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Filter.db 0/1436005
> >    outbrain_kvdb:
> > /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-123-Data.db 0/704736670
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Index.db
> > 0/230280
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Filter.db
> > 0/20605
> >    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-21-Data.db
> > 0/202808000
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Index.db
> > 0/23912
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Filter.db
> > 0/20605
> >    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-22-Data.db
> > 0/21434973
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Index.db
> > 0/965028
> >    Keyspace1:
> /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Filter.db
> > 0/20605
> >    Keyspace1: /outbrain/cassandra/data/Keyspace1/Standard1-tmp-23-Data.db
> > 0/865380205
> >
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Mime
View raw message