incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Burruss <>
Subject RE: node repair
Date Mon, 22 Mar 2010 16:53:37 GMT
it's very possible if i thought it wasn't working.  is there a delay between compation and
streaming?  maybe i didn't see any activity and assumed it was finished.  i am fairly certain
that if i had seen streaming action via JMX i would not have restarted.  and i know i didn't
see any compaction.

From: Stu Hood []
Sent: Monday, March 22, 2010 7:08 AM
Subject: RE: node repair

Hey Todd,

Repair involves 2 major compactions in addition to the streaming. More information is logged
about the compactions and repair when you are using DEBUG.

Do you think you might have restarted the node being repaired during the streaming process?
I'm not sure we have good handling for that case.


-----Original Message-----
From: "Todd Burruss" <>
Sent: Sunday, March 21, 2010 3:43pm
To: "" <>
Subject: RE: node repair

while preparing a test to capture logs i decided to not let the data set get too big and i
did see it finish.  i only had 2gb this go around, but i had about 120+gb before so maybe
it was just taking a long time.

that being said, i believe we need something in the logs, nodetool, and/or JMX that indicates
a repair is happening and it is finished.  i see messages about streaming in JMX, but the
current status for the crashed/repaired ( host is "Sending a stream initiate
message to /".  but nodetool streams reports:

[bburruss@kv-app03 ~]$ ~/cassandra/bin/nodetool -host localhost -port 9000 streams
Mode: Normal
 Nothing streaming to /
Not receiving any streams.
[bburruss@kv-app03 ~]$

... so i guess it is working, but maybe just some confusion with messaging .. i can't explain
why nodetool ring shows the sizes it does below except for read repair ... i'll keep an eye
out for it again and try it again with more data.

From: Stu Hood []
Sent: Sunday, March 21, 2010 12:08 PM
Subject: RE: node repair

If you have debug logs from the run, would you mind opening a JIRA describing the problem?

-----Original Message-----
From: "Todd Burruss" <>
Sent: Sunday, March 21, 2010 1:30pm
To: "Todd Burruss" <>, "" <>
Subject: RE: node repair

one last comment about thesting this is i stopped all the servers, wiped their data and restarted.
 allowed each node to get about 15gb on them, then repeated the test.  the nodetool repair
does not repair the crashed node.

the only minorly interesting thing about my cluster is that i use random partitioner and assigned
a token to each node.

From: Todd Burruss
Sent: Saturday, March 20, 2010 6:48 PM
To: Todd Burruss;
Subject: RE: node repair

fyi ... i just compacted and node 105 is definitely not being repaired
From: Todd Burruss
Sent: Saturday, March 20, 2010 12:34 PM
Subject: RE: node repair

same IP, same token.  i'm trying Handling Failure, #3.

it is running, a part of the ring, and seems to be handling reads/writes, but does not appear
to have received a copy of its data (the last node below).  i've searched the all logs for
ERRORs but there are none.  i will compact the other nodes, but i don't think it will make
a difference.

[bburruss@kv-app05 ~]$ ~/cassandra/bin/nodetool -h localhost -p 9000 ring
Address       Status     Load          Range                                      Ring
                                       170141183460469231731687303715884105728         130.22 GB     42535295865117307932921825928971026431     |<--|         131.03 GB     85070591730234615865843651857942052863     |   |         125.7 GB      127605887595351923798765477786913079295    |   |         65.62 GB      170141183460469231731687303715884105728    |-->|

From: Jonathan Ellis []
Sent: Saturday, March 20, 2010 11:23 AM
Subject: Re: node repair

if you bring up a new node w/ a different ip but the same token, it
will confuse things. "handling failure" section
covers best practices here.

On Sat, Mar 20, 2010 at 11:51 AM, Todd Burruss <> wrote:
> i had a node fail, lost all data.  so i brought it back up fresh, but assigned it the
same token in storage-conf.xml.  then ran nodetool repair.
> all compactions have finished, no streams are happening.  nothing.  so i did it again.
 same thing.  i don't think its working.  is there a log message i can search for?  INFO is
my log level.  i could try it again with debug i suppose.
> thx
View raw message