Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5D0999E53 for ; Thu, 12 Apr 2012 04:06:38 +0000 (UTC) Received: (qmail 43464 invoked by uid 500); 12 Apr 2012 04:06:36 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 43082 invoked by uid 500); 12 Apr 2012 04:06:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 43034 invoked by uid 99); 12 Apr 2012 04:06:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Apr 2012 04:06:33 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of berrytemk@gmail.com designates 209.85.215.44 as permitted sender) Received: from [209.85.215.44] (HELO mail-lpp01m010-f44.google.com) (209.85.215.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Apr 2012 04:06:27 +0000 Received: by lagj5 with SMTP id j5so1330149lag.31 for ; Wed, 11 Apr 2012 21:06:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=bKR+RMY2B80FRF2mg4UpDwkA5uRvFGGtldO1i+kqpsc=; b=d3pHTB9dCN421CLHJDdKqlieoJWppmIK5qZTscPiB0G9ur0bbIGjyTtovAZAHSAB9v WEQYV1ZRliPyoxRqhdMS6yWp5uh5PwPY1Q614AuhhsT+WgS+ZUD0bZYn8ueT+yKNxk6H rCaGfO+HrO4MBuSzl1H1k2oHxIFPKm2kepaT1b2M/4vaGKt3mrlGNlHWVrV5EzFF61oV rAi81yLEbe6DpcHSGtSwi/K0yQumieA5mtU1NLyjoDC1EBrn+mtQa+DrzWgshPNUfd/I QPFcI9uVuhijs4znHUY5eHqIx5ENSvpc0sVhTHC1aMwU94hp+RB/4gtO3uA2zrMMctGX USjw== MIME-Version: 1.0 Received: by 10.152.113.229 with SMTP id jb5mr671248lab.45.1334203566597; Wed, 11 Apr 2012 21:06:06 -0700 (PDT) Received: by 10.112.54.9 with HTTP; Wed, 11 Apr 2012 21:06:06 -0700 (PDT) In-Reply-To: <399A1DFA-D0AC-4355-B2AC-3A0DFCFB6ADE@thelastpickle.com> References: <4F8446F0.7070906@4friends.od.ua> <4F84639D.7050304@4friends.od.ua> <399A1DFA-D0AC-4355-B2AC-3A0DFCFB6ADE@thelastpickle.com> Date: Thu, 12 Apr 2012 00:06:06 -0400 Message-ID: Subject: Re: Repair Process Taking too long From: Frank Ng To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d04088d8ba5253804bd73793c --f46d04088d8ba5253804bd73793c Content-Type: text/plain; charset=ISO-8859-1 Thank you for confirming that the per node data size is most likely causing the long repair process. I have tried a repair on smaller column families and it was significantly faster. On Wed, Apr 11, 2012 at 9:55 PM, aaron morton wrote: > If you have 1TB of data it will take a long time to repair. Every bit of > data has to be read and a hash generated. This is one of the reasons we > often suggest that around 300 to 400Gb per node is a good load in the > general case. > > Look at nodetool compactionstats .Is there a validation compaction running > ? If so it is still building the merkle hash tree. > > Look at nodetool netstats . Is it streaming data ? If so all hash trees > have been calculated. > > Cheers > > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 12/04/2012, at 2:16 AM, Frank Ng wrote: > > Can you expand further on your issue? Were you using Random Patitioner? > > thanks > > On Tue, Apr 10, 2012 at 5:35 PM, David Leimbach wrote: > >> I had this happen when I had really poorly generated tokens for the ring. >> Cassandra seems to accept numbers that are too big. You get hot spots >> when you think you should be balanced and repair never ends (I think there >> is a 48 hour timeout). >> >> >> On Tuesday, April 10, 2012, Frank Ng wrote: >> >>> I am not using tier-sized compaction. >>> >>> >>> On Tue, Apr 10, 2012 at 12:56 PM, Jonathan Rhone wrote: >>> >>>> Data size, number of nodes, RF? >>>> >>>> Are you using size-tiered compaction on any of the column families that >>>> hold a lot of your data? >>>> >>>> Do your cassandra logs say you are streaming a lot of ranges? >>>> zgrep -E "(Performing streaming repair|out of sync)" >>>> >>>> >>>> On Tue, Apr 10, 2012 at 9:45 AM, Igor wrote: >>>> >>>>> On 04/10/2012 07:16 PM, Frank Ng wrote: >>>>> >>>>> Short answer - yes. >>>>> But you are asking wrong question. >>>>> >>>>> >>>>> I think both processes are taking a while. When it starts up, >>>>> netstats and compactionstats show nothing. Anyone out there successfully >>>>> using ext3 and their repair processes are faster than this? >>>>> >>>>> On Tue, Apr 10, 2012 at 10:42 AM, Igor wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> You can check with nodetool which part of repair process is slow - >>>>>> network streams or verify compactions. use nodetool netstats or >>>>>> compactionstats. >>>>>> >>>>>> >>>>>> On 04/10/2012 05:16 PM, Frank Ng wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I am on Cassandra 1.0.7. My repair processes are taking over 30 >>>>>>> hours to complete. Is it normal for the repair process to take this long? >>>>>>> I wonder if it's because I am using the ext3 file system. >>>>>>> >>>>>>> thanks >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Jonathan Rhone >>>> Software Engineer >>>> >>>> *TinyCo* >>>> 800 Market St., Fl 6 >>>> San Francisco, CA 94102 >>>> www.tinyco.com >>>> >>>> >>> > > --f46d04088d8ba5253804bd73793c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thank you for confirming that the per node data size is most likely causing= the long repair process. =A0I have tried a repair on smaller column famili= es and it was significantly faster.

On We= d, Apr 11, 2012 at 9:55 PM, aaron morton <aaron@thelastpickle.com> wrote= :
If you h= ave 1TB of data it will take a long time to repair. Every bit of data has t= o be read and a hash generated. This is one of the reasons we often suggest= that around 300 to 400Gb per node is a good load in the general case.=A0
Look at nodetool compactionstats .Is there a validation comp= action running ? If so it is still building the merkle =A0hash tree.=A0

Look at nodetool netstats . Is it streaming data ? If= so all hash trees have been calculated.=A0

Cheers


<= div style=3D"word-wrap:break-word">
-----------------
Aaron Morton
Freelance Deve= loper
@aaronmorton

On 12/04/2012, at 2:16 AM, Frank Ng wrote:

Can you expand further on your issue? Were you using Rando= m Patitioner?

thanks

On Tue, Apr 1= 0, 2012 at 5:35 PM, David Leimbach <leimy2k@gmail.com> wrote= :
I had this happen when I had really poorly g= enerated tokens for the ring. =A0Cassandra seems to accept numbers that are= too big. =A0You get hot spots when you think you should be balanced and re= pair never ends (I think there is a 48 hour timeout).


On Tuesday, April 10, 2012, Frank Ng wrote:
I am not using tier-sized compaction.


On Tue, Apr 10, 2012 at 12:56 PM, Jonathan Rhone <r= hone@tinyco.com> wrote:
Data size, num= ber of nodes, RF?

Are you using size-tiered compaction on any of the column fa= milies that hold a lot of your data?

Do your cassand= ra logs say you are streaming a lot of ranges?
zgrep -E "(Performing strea= ming repair|out of sync)"=A0


On Tue, Apr 1= 0, 2012 at 9:45 AM, Igor <igor@4friends.od.ua&g= t; wrote:
=20 =20 =20
On 04/10/2012 07:16 PM, Frank Ng wrote:

Short answer - yes.
But you are asking wrong question.


I think both processes are taking a while.=A0= When it starts up, netstats and compactionstats show nothing.=A0 Anyone out there successfully using ext3 and their repair processes are faster than this?

On Tue, Apr 10, 2012 at 10:42 AM, Igor <igo= r@4friends.od.ua> wrote:
Hi

You can check with nodetool =A0which part of repair process is slow - network streams or verify compactions. use nodetool netstats or compactionstats.


On 04/10/2012 05:16 PM, Frank Ng wrote:
Hello,

I am on Cassandra 1.0.7. =A0My repair processes are taking over 30 hours to complete. =A0Is it normal for the repair process to take this long? =A0I wonder if it's because = I am using the ext3 file system.

thanks






<= font color=3D"#888888">--
Jonathan Rhone
Software Engineer
=

= TinyCo
800 Market St., Fl = 6




--f46d04088d8ba5253804bd73793c--