incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joost Ouwerkerk <jo...@openplaces.org>
Subject Re: get_range_slices confused about token ranges after decommissioning a node
Date Tue, 22 Jun 2010 14:39:04 GMT
I don't mind missing data for a few hours, it's the weird behaviour of
get_range_slices that's bothering me.  I added some logging to
ColumnFamilyRecordReader to see what's going on:

Split startToken=67160993471237854630929198835217410155,
endToken=68643623863384825230116928934887817211

...

Getting batch for range: 67965855060996012099315582648654139032 to
68643623863384825230116928934887817211

Token for last row is: 50448492574454416067449808504057295946

Getting batch for range: 50448492574454416067449808504057295946 to
68643623863384825230116928934887817211

...


Notice how the get_range_slices response is invalid since it returns an
out-of-range row.  This poisons the batching loop and causes the task to
spin out of control.

/joost

On Tue, Jun 22, 2010 at 9:09 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

> What I would expect to have happen is for the removed node to
> disappear from the ring and for nodes that are supposed to get more
> data to start streaming it over.  I would expect it to be hours before
> any new data started appearing anywhere when you are anticompacting
> 80+GB prior to the streaming part.
> http://wiki.apache.org/cassandra/Streaming
>
> On Tue, Jun 22, 2010 at 12:57 AM, Joost Ouwerkerk <joost@openplaces.org>
> wrote:
> > Yes, although "forget" implies that we once knew we were supposed to do
> so.
> > Given the following before-and-after states, on which nodes are we
> supposed
> > to run repair?  Should the cluster be restarted?  Is there anything else
> we
> > should be doing, or not doing?
> >
> > 1. Node is down due to hardware failure
> >
> > 192.168.1.104 Up         111.75 GB
> > 8954799129498380617457226511362321354      |   ^
> > 192.168.1.106 Up         113.25 GB
> > 17909598258996761234914453022724642708     v   |
> > 192.168.1.107 Up         75.65 GB
> > 22386997823745951543643066278405803385     |   ^
> > 192.168.1.108 Down    75.77 GB
> > 26864397388495141852371679534086964062     v   |
> > 192.168.1.109 Up         76.14 GB
> > 35819196517993522469828906045449285416     |   ^
> > 192.168.1.110 Up         75.9 GB
> > 40296596082742712778557519301130446093     v   |
> > 192.168.1.111 Up         95.21 GB
> > 49251395212241093396014745812492767447     |   ^
> >
> > 2. nodetool removetoken 26864397388495141852371679534086964062
> >
> > 192.168.1.104 Up         111.75 GB
> > 8954799129498380617457226511362321354      |   ^
> > 192.168.1.106 Up         113.25 GB
> > 17909598258996761234914453022724642708     v   |
> > 192.168.1.107 Up         75.65 GB
> > 22386997823745951543643066278405803385     |   ^
> > 192.168.1.109 Up         76.14 GB
> > 35819196517993522469828906045449285416     |   ^
> > 192.168.1.110 Up         75.9 GB
> > 40296596082742712778557519301130446093     v   |
> > 192.168.1.111 Up         95.21 GB
> > 49251395212241093396014745812492767447     |   ^
> >
> > At this point we're expecting 192.168.1.107 to pick up the slack for the
> > removed token, and for 192.168.1.109 and/or 192.168.1.110 to start
> streaming
> > data to 192.168.1.107 since they are holding the replicated data for that
> > range.
> >
> > 3. nodetool repair ?
> >
> > On Tue, Jun 22, 2010 at 12:03 AM, Benjamin Black <b@b3k.us> wrote:
> >>
> >> Did you forget to run repair?
> >>
> >> On Mon, Jun 21, 2010 at 7:02 PM, Joost Ouwerkerk <joost@openplaces.org>
> >> wrote:
> >> > I believe we did nodetool removetoken on nodes that were already down
> >> > (due
> >> > to hardware failure), but I will check to make sure. We're running
> >> > Cassandra
> >> > 0.6.2.
> >> >
> >> > On Mon, Jun 21, 2010 at 9:59 PM, Joost Ouwerkerk <
> joost@openplaces.org>
> >> > wrote:
> >> >>
> >> >> Greg, can you describe the steps we took to decommission the nodes?
> >> >>
> >> >> ---------- Forwarded message ----------
> >> >> From: Rob Coli <rcoli@digg.com>
> >> >> Date: Mon, Jun 21, 2010 at 8:08 PM
> >> >> Subject: Re: get_range_slices confused about token ranges after
> >> >> decommissioning a node
> >> >> To: user@cassandra.apache.org
> >> >>
> >> >>
> >> >> On 6/21/10 4:57 PM, Joost Ouwerkerk wrote:
> >> >>>
> >> >>> We're seeing very strange behaviour after decommissioning a node:
> when
> >> >>> requesting a get_range_slices with a KeyRange by token, we are
> getting
> >> >>> back tokens that are out of range.
> >> >>
> >> >> What sequence of actions did you take to "decommission" the node?
> What
> >> >> version of Cassandra are you running?
> >> >>
> >> >> =Rob
> >> >>
> >> >
> >> >
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Mime
View raw message