incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Repair Process Taking too long
Date Tue, 22 May 2012 09:16:57 GMT
It repairs the ranges they have in common. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 20/05/2012, at 4:05 PM, Raj N wrote:

> Can I infer from this that if I have 3 replicas, then running repair without -pr won
1 node will repair the other 2 replicas as well.
> 
> -Raj
> 
> On Sat, Apr 14, 2012 at 2:54 AM, Zhu Han <hanzhu@nutstore.net> wrote:
> 
> On Sat, Apr 14, 2012 at 1:57 PM, Igor <igor@4friends.od.ua> wrote:
> Hi!
> 
> What is the difference between 'repair' and '-pr repair'? Simple repair touch all token
ranges (for all nodes) and -pr touch only range for which given node responsible?
> 
> 
> -pr only touches the primary range of the node.  If you executes -pr against all nodes
in replica groups,  then all ranges are repaired.
> 
> 
> On 04/12/2012 05:59 PM, Sylvain Lebresne wrote:
> On Thu, Apr 12, 2012 at 4:06 PM, Frank Ng<buzztemk@gmail.com>  wrote:
> I also noticed that if I use the -pr option, the repair process went down
> from 30 hours to 9 hours.  Is the -pr option safe to use if I want to run
> repair processes in parallel on nodes that are not replication peers?
> There is pretty much two use case for repair:
> 1) to rebuild a node: if say a node has lost some data due to a hard
> drive corruption or the like and you want to to rebuild what's missing
> 2) the periodic repairs to avoid problem with deleted data coming back
> from the dead (basically:
> http://wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair)
> 
> In case 1) you want to run 'nodetool repair' (without -pr) against the
> node to rebuild.
> In case 2) (which I suspect is the case your talking now), you *want*
> to use 'nodetool repair -pr' on *every* node of the cluster. I.e.
> that's the most efficient way to do it. The only reason not to use -pr
> in this case would be that it's not available because you're using an
> old version of Cassandra. And yes, it's is safe to run with -pr in
> parallel on nodes that are not replication peers.
> 
> --
> Sylvain
> 
> 
> thanks
> 
> 
> On Thu, Apr 12, 2012 at 12:06 AM, Frank Ng<berrytemk@gmail.com>  wrote:
> Thank you for confirming that the per node data size is most likely
> causing the long repair process.  I have tried a repair on smaller column
> families and it was significantly faster.
> 
> On Wed, Apr 11, 2012 at 9:55 PM, aaron morton<aaron@thelastpickle.com>
> wrote:
> If you have 1TB of data it will take a long time to repair. Every bit of
> data has to be read and a hash generated. This is one of the reasons we
> often suggest that around 300 to 400Gb per node is a good load in the
> general case.
> 
> Look at nodetool compactionstats .Is there a validation compaction
> running ? If so it is still building the merkle  hash tree.
> 
> Look at nodetool netstats . Is it streaming data ? If so all hash trees
> have been calculated.
> 
> Cheers
> 
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 12/04/2012, at 2:16 AM, Frank Ng wrote:
> 
> Can you expand further on your issue? Were you using Random Patitioner?
> 
> thanks
> 
> On Tue, Apr 10, 2012 at 5:35 PM, David Leimbach<leimy2k@gmail.com>
> wrote:
> I had this happen when I had really poorly generated tokens for the
> ring.  Cassandra seems to accept numbers that are too big.  You get hot
> spots when you think you should be balanced and repair never ends (I think
> there is a 48 hour timeout).
> 
> 
> On Tuesday, April 10, 2012, Frank Ng wrote:
> I am not using tier-sized compaction.
> 
> 
> On Tue, Apr 10, 2012 at 12:56 PM, Jonathan Rhone<rhone@tinyco.com>
> wrote:
> Data size, number of nodes, RF?
> 
> Are you using size-tiered compaction on any of the column families
> that hold a lot of your data?
> 
> Do your cassandra logs say you are streaming a lot of ranges?
> zgrep -E "(Performing streaming repair|out of sync)"
> 
> 
> On Tue, Apr 10, 2012 at 9:45 AM, Igor<igor@4friends.od.ua>  wrote:
> On 04/10/2012 07:16 PM, Frank Ng wrote:
> 
> Short answer - yes.
> But you are asking wrong question.
> 
> 
> I think both processes are taking a while.  When it starts up,
> netstats and compactionstats show nothing.  Anyone out there successfully
> using ext3 and their repair processes are faster than this?
> 
> On Tue, Apr 10, 2012 at 10:42 AM, Igor<igor@4friends.od.ua>  wrote:
> Hi
> 
> You can check with nodetool  which part of repair process is slow -
> network streams or verify compactions. use nodetool netstats or
> compactionstats.
> 
> 
> On 04/10/2012 05:16 PM, Frank Ng wrote:
> Hello,
> 
> I am on Cassandra 1.0.7.  My repair processes are taking over 30
> hours to complete.  Is it normal for the repair process to take this long?
>  I wonder if it's because I am using the ext3 file system.
> 
> thanks
> 
> 
> 
> 
> --
> Jonathan Rhone
> Software Engineer
> 
> TinyCo
> 800 Market St., Fl 6
> San Francisco, CA 94102
> www.tinyco.com
> 
> 
> 
> 
> 


Mime
View raw message