incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raj N <raj.cassan...@gmail.com>
Subject Re: Repair Process Taking too long
Date Sun, 20 May 2012 04:05:11 GMT
Can I infer from this that if I have 3 replicas, then running repair
without -pr won 1 node will repair the other 2 replicas as well.

-Raj

On Sat, Apr 14, 2012 at 2:54 AM, Zhu Han <hanzhu@nutstore.net> wrote:

>
> On Sat, Apr 14, 2012 at 1:57 PM, Igor <igor@4friends.od.ua> wrote:
>
>> Hi!
>>
>> What is the difference between 'repair' and '-pr repair'? Simple repair
>> touch all token ranges (for all nodes) and -pr touch only range for which
>> given node responsible?
>>
>>
> -pr only touches the primary range of the node.  If you executes -pr
> against all nodes in replica groups,  then all ranges are repaired.
>
>>
>>
>> On 04/12/2012 05:59 PM, Sylvain Lebresne wrote:
>>
>>> On Thu, Apr 12, 2012 at 4:06 PM, Frank Ng<buzztemk@gmail.com>  wrote:
>>>
>>>> I also noticed that if I use the -pr option, the repair process went
>>>> down
>>>> from 30 hours to 9 hours.  Is the -pr option safe to use if I want to
>>>> run
>>>> repair processes in parallel on nodes that are not replication peers?
>>>>
>>> There is pretty much two use case for repair:
>>> 1) to rebuild a node: if say a node has lost some data due to a hard
>>> drive corruption or the like and you want to to rebuild what's missing
>>> 2) the periodic repairs to avoid problem with deleted data coming back
>>> from the dead (basically:
>>> http://wiki.apache.org/**cassandra/Operations#**
>>> Frequency_of_nodetool_repair<http://wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair>
>>> )
>>>
>>> In case 1) you want to run 'nodetool repair' (without -pr) against the
>>> node to rebuild.
>>> In case 2) (which I suspect is the case your talking now), you *want*
>>> to use 'nodetool repair -pr' on *every* node of the cluster. I.e.
>>> that's the most efficient way to do it. The only reason not to use -pr
>>> in this case would be that it's not available because you're using an
>>> old version of Cassandra. And yes, it's is safe to run with -pr in
>>> parallel on nodes that are not replication peers.
>>>
>>> --
>>> Sylvain
>>>
>>>
>>>  thanks
>>>>
>>>>
>>>> On Thu, Apr 12, 2012 at 12:06 AM, Frank Ng<berrytemk@gmail.com>  wrote:
>>>>
>>>>> Thank you for confirming that the per node data size is most likely
>>>>> causing the long repair process.  I have tried a repair on smaller
>>>>> column
>>>>> families and it was significantly faster.
>>>>>
>>>>> On Wed, Apr 11, 2012 at 9:55 PM, aaron morton<aaron@thelastpickle.com*
>>>>> *>
>>>>> wrote:
>>>>>
>>>>>> If you have 1TB of data it will take a long time to repair. Every
bit
>>>>>> of
>>>>>> data has to be read and a hash generated. This is one of the reasons
>>>>>> we
>>>>>> often suggest that around 300 to 400Gb per node is a good load in
the
>>>>>> general case.
>>>>>>
>>>>>> Look at nodetool compactionstats .Is there a validation compaction
>>>>>> running ? If so it is still building the merkle  hash tree.
>>>>>>
>>>>>> Look at nodetool netstats . Is it streaming data ? If so all hash
>>>>>> trees
>>>>>> have been calculated.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> -----------------
>>>>>> Aaron Morton
>>>>>> Freelance Developer
>>>>>> @aaronmorton
>>>>>> http://www.thelastpickle.com
>>>>>>
>>>>>> On 12/04/2012, at 2:16 AM, Frank Ng wrote:
>>>>>>
>>>>>> Can you expand further on your issue? Were you using Random
>>>>>> Patitioner?
>>>>>>
>>>>>> thanks
>>>>>>
>>>>>> On Tue, Apr 10, 2012 at 5:35 PM, David Leimbach<leimy2k@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I had this happen when I had really poorly generated tokens for
the
>>>>>>> ring.  Cassandra seems to accept numbers that are too big.  You
get
>>>>>>> hot
>>>>>>> spots when you think you should be balanced and repair never
ends (I
>>>>>>> think
>>>>>>> there is a 48 hour timeout).
>>>>>>>
>>>>>>>
>>>>>>> On Tuesday, April 10, 2012, Frank Ng wrote:
>>>>>>>
>>>>>>>> I am not using tier-sized compaction.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 10, 2012 at 12:56 PM, Jonathan Rhone<rhone@tinyco.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Data size, number of nodes, RF?
>>>>>>>>>
>>>>>>>>> Are you using size-tiered compaction on any of the column
families
>>>>>>>>> that hold a lot of your data?
>>>>>>>>>
>>>>>>>>> Do your cassandra logs say you are streaming a lot of
ranges?
>>>>>>>>> zgrep -E "(Performing streaming repair|out of sync)"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Apr 10, 2012 at 9:45 AM, Igor<igor@4friends.od.ua>
 wrote:
>>>>>>>>>
>>>>>>>>>> On 04/10/2012 07:16 PM, Frank Ng wrote:
>>>>>>>>>>
>>>>>>>>>> Short answer - yes.
>>>>>>>>>> But you are asking wrong question.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think both processes are taking a while.  When
it starts up,
>>>>>>>>>> netstats and compactionstats show nothing.  Anyone
out there
>>>>>>>>>> successfully
>>>>>>>>>> using ext3 and their repair processes are faster
than this?
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 10, 2012 at 10:42 AM, Igor<igor@4friends.od.ua>
>>>>>>>>>>  wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> You can check with nodetool  which part of repair
process is
>>>>>>>>>>> slow -
>>>>>>>>>>> network streams or verify compactions. use nodetool
netstats or
>>>>>>>>>>> compactionstats.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 04/10/2012 05:16 PM, Frank Ng wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> I am on Cassandra 1.0.7.  My repair processes
are taking over 30
>>>>>>>>>>>> hours to complete.  Is it normal for the
repair process to take
>>>>>>>>>>>> this long?
>>>>>>>>>>>>  I wonder if it's because I am using the
ext3 file system.
>>>>>>>>>>>>
>>>>>>>>>>>> thanks
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jonathan Rhone
>>>>>>>>> Software Engineer
>>>>>>>>>
>>>>>>>>> TinyCo
>>>>>>>>> 800 Market St., Fl 6
>>>>>>>>> San Francisco, CA 94102
>>>>>>>>> www.tinyco.com
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>
>

Mime
View raw message