cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raj N <raj.cassan...@gmail.com>
Subject Re: nodetool repair taking forever
Date Fri, 25 May 2012 22:04:31 GMT
Thanks for the reply Aaron. By compaction being on, do you mean if run
nodetool compact, then the answer is no. I haven't set any explicit
compaction_thresholds which means it should be using the default, min 4 and
max 32. Having said that to solve the problem, I just did a full cluster
restart and ran nodetool repair again. The entire cluster of 6 nodes was
repaired in 10 hours. I am also contemplating since all the 6 nodes are
replicas of each other, do I even need to run repair on all the nodes.
Wouldn't running it on the first node suffice since it will repair all the
ranges its responsible for(which is everything). So unless I upgrade to
1.0.+, where I can use the -pr option is it advisable to just run repair on
the first node.

-Raj

On Tue, May 22, 2012 at 5:05 AM, aaron morton <aaron@thelastpickle.com>wrote:

> I also dont understand if all these nodes are replicas of each other why
> is that the first node has almost double the data.
>
> Have you performed any token moves ? Old data is not deleted unless you
> run nodetool cleanup.
> Another possibility is things like a lot of hints. Admittedly it would
> have to be a *lot* of hints.
> The third is that compaction has fallen behind.
>
> This week its even worse, the nodetool repair has been running for the
> last 15 hours just on the first node and when I run nodetool
> compactionstats I constantly see this -
>
> pending tasks: 3
>
> First check the logs for errors.
>
> Repair will first calculate the differences, you can see this as a
> validation compaction in nodetool compactionstats.
> Then it will stream the data, you can watch that with nodetool netstats.
>
> Try to work out which part is taking the most time.   15 hours for 50Gb
> sounds like a long time (btw do you have compaction on ?)
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/05/2012, at 3:14 AM, Raj N wrote:
>
> Hi experts,
>
> I have a 6 node cluster spread across 2 DCs.
>
>     DC          Rack        Status State   Load            Owns    Token
>
>  113427455640312814857969558651062452225
>     DC1         RAC13       Up     Normal  95.98 GB        33.33%  0
>     DC2         RAC5        Up     Normal  50.79 GB        0.00%   1
>     DC1         RAC18       Up     Normal  50.83 GB        33.33%
>  56713727820156407428984779325531226112
>     DC2         RAC7        Up     Normal  50.74 GB        0.00%
> 56713727820156407428984779325531226113
>     DC1         RAC19       Up     Normal  61.72 GB        33.33%
>  113427455640312814857969558651062452224
>     DC2         RAC9        Up     Normal  50.83 GB        0.00%
> 113427455640312814857969558651062452225
>
> They are all replicas of each other. All reads and writes are done at
> LOCAL_QUORUM. We are on Cassandra 0.8.4. I see that our weekend nodetool
> repair runs for more than 12 hours. Especially on the first one which has
> 96 GB data. Is this usual? We are using 500 GB SAS drives with ext4 file
> system. This gets worse every week. This week its even worse, the nodetool
> repair has been running for the last 15 hours just on the first node and
> when I run nodetool compactionstats I constantly see this -
>
> pending tasks: 3
>
> and nothing else. Looks like its just stuck. There's nothing substantial
> in the logs as well. I also dont understand if all these nodes are replicas
> of each other why is that the first node has almost double the data. Any
> help will be really appreciated.
>
> Thanks
> -Raj
>
>
>

Mime
View raw message