incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Ng <buzzt...@gmail.com>
Subject Re: Repair Process Taking too long
Date Thu, 12 Apr 2012 20:10:03 GMT
Thanks for the clarification.  I'm running repairs as in case 2 (to avoid
deleted data coming back).

On Thu, Apr 12, 2012 at 10:59 AM, Sylvain Lebresne <sylvain@datastax.com>wrote:

> On Thu, Apr 12, 2012 at 4:06 PM, Frank Ng <buzztemk@gmail.com> wrote:
> > I also noticed that if I use the -pr option, the repair process went down
> > from 30 hours to 9 hours.  Is the -pr option safe to use if I want to run
> > repair processes in parallel on nodes that are not replication peers?
>
> There is pretty much two use case for repair:
> 1) to rebuild a node: if say a node has lost some data due to a hard
> drive corruption or the like and you want to to rebuild what's missing
> 2) the periodic repairs to avoid problem with deleted data coming back
> from the dead (basically:
> http://wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair)
>
> In case 1) you want to run 'nodetool repair' (without -pr) against the
> node to rebuild.
> In case 2) (which I suspect is the case your talking now), you *want*
> to use 'nodetool repair -pr' on *every* node of the cluster. I.e.
> that's the most efficient way to do it. The only reason not to use -pr
> in this case would be that it's not available because you're using an
> old version of Cassandra. And yes, it's is safe to run with -pr in
> parallel on nodes that are not replication peers.
>
> --
> Sylvain
>
>
> >
> > thanks
> >
> >
> > On Thu, Apr 12, 2012 at 12:06 AM, Frank Ng <berrytemk@gmail.com> wrote:
> >>
> >> Thank you for confirming that the per node data size is most likely
> >> causing the long repair process.  I have tried a repair on smaller
> column
> >> families and it was significantly faster.
> >>
> >> On Wed, Apr 11, 2012 at 9:55 PM, aaron morton <aaron@thelastpickle.com>
> >> wrote:
> >>>
> >>> If you have 1TB of data it will take a long time to repair. Every bit
> of
> >>> data has to be read and a hash generated. This is one of the reasons we
> >>> often suggest that around 300 to 400Gb per node is a good load in the
> >>> general case.
> >>>
> >>> Look at nodetool compactionstats .Is there a validation compaction
> >>> running ? If so it is still building the merkle  hash tree.
> >>>
> >>> Look at nodetool netstats . Is it streaming data ? If so all hash trees
> >>> have been calculated.
> >>>
> >>> Cheers
> >>>
> >>>
> >>> -----------------
> >>> Aaron Morton
> >>> Freelance Developer
> >>> @aaronmorton
> >>> http://www.thelastpickle.com
> >>>
> >>> On 12/04/2012, at 2:16 AM, Frank Ng wrote:
> >>>
> >>> Can you expand further on your issue? Were you using Random Patitioner?
> >>>
> >>> thanks
> >>>
> >>> On Tue, Apr 10, 2012 at 5:35 PM, David Leimbach <leimy2k@gmail.com>
> >>> wrote:
> >>>>
> >>>> I had this happen when I had really poorly generated tokens for the
> >>>> ring.  Cassandra seems to accept numbers that are too big.  You get
> hot
> >>>> spots when you think you should be balanced and repair never ends (I
> think
> >>>> there is a 48 hour timeout).
> >>>>
> >>>>
> >>>> On Tuesday, April 10, 2012, Frank Ng wrote:
> >>>>>
> >>>>> I am not using tier-sized compaction.
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 10, 2012 at 12:56 PM, Jonathan Rhone <rhone@tinyco.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Data size, number of nodes, RF?
> >>>>>>
> >>>>>> Are you using size-tiered compaction on any of the column families
> >>>>>> that hold a lot of your data?
> >>>>>>
> >>>>>> Do your cassandra logs say you are streaming a lot of ranges?
> >>>>>> zgrep -E "(Performing streaming repair|out of sync)"
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Apr 10, 2012 at 9:45 AM, Igor <igor@4friends.od.ua>
wrote:
> >>>>>>>
> >>>>>>> On 04/10/2012 07:16 PM, Frank Ng wrote:
> >>>>>>>
> >>>>>>> Short answer - yes.
> >>>>>>> But you are asking wrong question.
> >>>>>>>
> >>>>>>>
> >>>>>>> I think both processes are taking a while.  When it starts
up,
> >>>>>>> netstats and compactionstats show nothing.  Anyone out there
> successfully
> >>>>>>> using ext3 and their repair processes are faster than this?
> >>>>>>>
> >>>>>>> On Tue, Apr 10, 2012 at 10:42 AM, Igor <igor@4friends.od.ua>
> wrote:
> >>>>>>>>
> >>>>>>>> Hi
> >>>>>>>>
> >>>>>>>> You can check with nodetool  which part of repair process
is slow
> -
> >>>>>>>> network streams or verify compactions. use nodetool
netstats or
> >>>>>>>> compactionstats.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 04/10/2012 05:16 PM, Frank Ng wrote:
> >>>>>>>>>
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> I am on Cassandra 1.0.7.  My repair processes are
taking over 30
> >>>>>>>>> hours to complete.  Is it normal for the repair
process to take
> this long?
> >>>>>>>>>  I wonder if it's because I am using the ext3 file
system.
> >>>>>>>>>
> >>>>>>>>> thanks
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Jonathan Rhone
> >>>>>> Software Engineer
> >>>>>>
> >>>>>> TinyCo
> >>>>>> 800 Market St., Fl 6
> >>>>>> San Francisco, CA 94102
> >>>>>> www.tinyco.com
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
> >
>

Mime
View raw message