cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Dusbabek <gdusba...@gmail.com>
Subject Re: nodetool loadbalance : Strerams Continue on Non Acceptance of New Token
Date Wed, 23 Jun 2010 12:40:02 GMT
On Tue, Jun 22, 2010 at 20:16, Arya Goudarzi <agoudarzi@gaiaonline.com> wrote:
> Hi,
>
> Please confirm if this is an issue and should be reported or I am doing something wrong.
I could not find anything relevant on JIRA:
>
> Playing with 0.7 nightly (today's build), I setup a 3 node cluster this way:
>
>  - Added one node;
>  - Loaded default schema with RF 1 from YAML using JMX;
>  - Loaded 2M keys using py_stress;
>  - Bootstrapped a second node;
>  - Cleaned up the first node;
>  - Bootstrapped a third node;
>  - Cleaned up the second node;
>
> I got the following ring:
>
> Address       Status     Load          Range                    
                 Ring
>                                       154293670372423273273390365393543806425
> 10.50.26.132  Up         518.63 MB     69164917636305877859094619660693892452
    |<--|
> 10.50.26.134  Up         234.8 MB      111685517405103688771527967027648896391
   |   |
> 10.50.26.133  Up         235.26 MB     154293670372423273273390365393543806425
   |-->|
>
> Now I ran:
>
> nodetool --host 10.50.26.132 loadbalance
>
> It's been going for a while. I checked the streams
>
> nodetool --host 10.50.26.134 streams
> Mode: Normal
> Not sending any streams.
> Streaming from: /10.50.26.132
>   Keyspace1: /var/lib/cassandra/data/Keyspace1/Standard1-tmp-d-3-Data.db/[(0,22206096),
(22206096,27271682)]
>   Keyspace1: /var/lib/cassandra/data/Keyspace1/Standard1-tmp-d-4-Data.db/[(0,15180462),
(15180462,18656982)]
>   Keyspace1: /var/lib/cassandra/data/Keyspace1/Standard1-tmp-d-5-Data.db/[(0,353139829),
(353139829,433883659)]
>   Keyspace1: /var/lib/cassandra/data/Keyspace1/Standard1-tmp-d-6-Data.db/[(0,366336059),
(366336059,450095320)]
>
> nodetool --host 10.50.26.132 streams
> Mode: Leaving: streaming data to other nodes
> Streaming to: /10.50.26.134
>   /var/lib/cassandra/data/Keyspace1/Standard1-d-48-Data.db/[(0,366336059), (366336059,450095320)]
> Not receiving any streams.
>
> These have been going for the past 2 hours.
>
> I see in the logs of the node with 134 IP address and I saw this:
>
> INFO [GOSSIP_STAGE:1] 2010-06-22 16:30:54,679 StorageService.java (line 603) Will not
change my token ownership to /10.50.26.132

A node will give this message when it sees another node (usually for
the first time) that is trying to claim the same token but whose
startup time is much earlier (i.e., this isn't a token replacement).
It would follow that you would see this during a rebalance.

>
> So, to my understanding from wikis loadbalance supposed to decommission and re-bootstrap
again by sending its tokens to other nodes and then bootstrap again. It's been stuck in streaming
for the past 2 hours and the size of ring has not changed. The log in the first node says
it has started streaming for the past hours:
>
> INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,255 StreamOut.java (line 72) Beginning transfer
process to /10.50.26.134 for ranges (154293670372423273273390365393543806425,69164917636305877859094619660693892452]
>  INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,255 StreamOut.java (line 82) Flushing memtables
for Keyspace1...
>  INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,266 StreamOut.java (line 128) Stream context
metadata [/var/lib/cassandra/data/Keyspace1/Standard1-d-48-Data.db/[(0,366336059), (366336059,450095320)]]
1 sstables.
>  INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,267 StreamOut.java (line 135) Sending a stream
initiate message to /10.50.26.134 ...
>  INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,267 StreamOut.java (line 140) Waiting for
transfer to /10.50.26.134 to complete
>  INFO [FLUSH-TIMER] 2010-06-22 17:36:53,370 ColumnFamilyStore.java (line 359) LocationInfo
has reached its threshold; switching in a fresh Memtable at CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1277249454413.log',
position=720)
>  INFO [FLUSH-TIMER] 2010-06-22 17:36:53,370 ColumnFamilyStore.java (line 622) Enqueuing
flush of Memtable(LocationInfo)@1637794189
>  INFO [FLUSH-WRITER-POOL:1] 2010-06-22 17:36:53,370 Memtable.java (line 149) Writing
Memtable(LocationInfo)@1637794189
>  INFO [FLUSH-WRITER-POOL:1] 2010-06-22 17:36:53,528 Memtable.java (line 163) Completed
flushing /var/lib/cassandra/data/system/LocationInfo-d-9-Data.db
>  INFO [MEMTABLE-POST-FLUSHER:1] 2010-06-22 17:36:53,529 ColumnFamilyStore.java (line
374) Discarding 1000
>
>
> Nothing more after this line.
>
> Am I doing something wrong?

If the output you get from `nodetool streams` isn't changing, then I'd
say we have a bug.  You're data sizes weren't that large--I'd expect 2
hrs would be more than enough time.

I've created https://issues.apache.org/jira/browse/CASSANDRA-1221 to
track this problem.

Gary.

>
>
> Best Regards,
> -Arya
>
>

Mime
View raw message