cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Kania <>
Subject Re: Nodetool repair inconsistencies
Date Wed, 08 Jun 2016 17:11:43 GMT
Hi Paul,
I have tried running 'nodetool compact' and the situation remains the same after I deleted
the files that caused 'nodetool compact' to generate an exception in the first place.
My concern is that if I delete some sstable sets from a directory or even if I completely
eliminate the sstables in a directory on one machine, run 'nodetool repair' followed by 'nodetool
compact', that directory remains empty. My understanding has been that these equivalently
named directories should contain roughly the same amount of content.

      From: Paul Fife <>
 To:; Jason Kania <> 
 Sent: Wednesday, June 8, 2016 12:55 PM
 Subject: Re: Nodetool repair inconsistencies
Hi Jason -
Did you run a major compaction after the repair completed? Do you have other reasons besides
the number/size of sstables to believe all nodes don't have a copy of the current data at
the end of the repair operation?
On Wed, Jun 8, 2016 at 8:12 AM, Jason Kania <> wrote:

Hi Romain,
The problem is that there is no error to share. I am focusing on the inconsistency that when
I run nodetool repair, get no errors and yet the content in the same directory on the different
nodes is vastly different. This lack of an error is nature of my question, not the nodetool
compact error.
      From: Romain Hardouin <>
 To: "" <>; Jason Kania <>

 Sent: Wednesday, June 8, 2016 8:30 AM
 Subject: Re: Nodetool repair inconsistencies
Hi Jason,
It's difficult for the community to help you if you don't share the error ;-)What the logs
said when you ran a major compaction? (i.e. the first error you encountered) 

    Le Mercredi 8 juin 2016 3h34, Jason Kania <> a écrit :

 I am running a 3 node cluster of 3.0.6 instances and encountered an error when running nodetool
compact. I then ran nodetool repair. No errors were returned.
I then attempted to run nodetool compact again, but received the same error so the repair
made no correction and reported no errors.
After that, I moved the problematic files out of the directory, restarted cassandra and attempted
the repair again. The repair again completed without errors, however, no files were added
to the directory that had contained the corrupt files. So nodetool repair does not seem to
be making actual repairs.
I started looking around and numerous directories have vastly different amounts of content
across the 3 nodes. There are 3 replicas so I would expect to find similar amounts of content
in the same data directory on the different nodes.

Is there any way to dig deeper into this? I don't want to be caught because replication/repair
is silently failing. I noticed that there is always an "some repair failed" amongst the repair
output but that is so completely unhelpful and has always been present.




View raw message