Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 130DA17FC0 for ; Fri, 26 Sep 2014 18:21:16 +0000 (UTC) Received: (qmail 99287 invoked by uid 500); 26 Sep 2014 18:21:13 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 99253 invoked by uid 500); 26 Sep 2014 18:21:13 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 99243 invoked by uid 99); 26 Sep 2014 18:21:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Sep 2014 18:21:13 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jonathan.haddad@gmail.com designates 209.85.212.182 as permitted sender) Received: from [209.85.212.182] (HELO mail-wi0-f182.google.com) (209.85.212.182) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Sep 2014 18:20:46 +0000 Received: by mail-wi0-f182.google.com with SMTP id d1so21363wiv.3 for ; Fri, 26 Sep 2014 11:20:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=0PYRPSDNBM6nbCvQhVvhztbNCeS6QofTqbkl+vIn46I=; b=ogSylHAjiI1Xo42LE5Hva1wKEOcaAhWXAlY0ilylXRnUb04HwySbYAjXsXonVoK1Cb 9rI/jd/zvvFpkVTd+DgEiZwGzM+vNPBrhJD7/A/M183BwU6mmlmnDUKc7E+cNklvaRSq P0sKJ4lgUtNuL4+MtLBO+o6G5H+O2txXLCAiheOWQJ4d1yyeWTozyLd1awdvOSHb61RK qN0E27ZtUkbhuPIL9ZDEGYqfOxIlfTRtpgZqAiaX+vpXPG13nRqJhekheyQSiYJBxKJA 0vbTGqrAPwKv5mdpbUdYQgtRUbAL1l7zy+yvPMQPNV9ViMkiSE5jAOJbRRRKsZyEEpp4 cNhA== MIME-Version: 1.0 X-Received: by 10.194.95.8 with SMTP id dg8mr26496969wjb.1.1411755645975; Fri, 26 Sep 2014 11:20:45 -0700 (PDT) Sender: jonathan.haddad@gmail.com Received: by 10.216.168.68 with HTTP; Fri, 26 Sep 2014 11:20:45 -0700 (PDT) In-Reply-To: References: Date: Fri, 26 Sep 2014 11:20:45 -0700 X-Google-Sender-Auth: Z2BQCm9BmLGE1-bKc553YrrTe-c Message-ID: Subject: Re: Repair taking long time From: Jonathan Haddad To: "user@cassandra.apache.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Well, in that case, you may want to roll your own script for doing constant repairs of your cluster, and extend your gc grace seconds so you can repair the whole cluster before the tombstones are cleared. On Fri, Sep 26, 2014 at 11:15 AM, Gene Robichaux wrote: > Using their community edition......no support (yet!) :( > > Gene Robichaux > Manager, Database Operations > Match.com > 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 > Phone: 214-576-3273 > > -----Original Message----- > From: jonathan.haddad@gmail.com [mailto:jonathan.haddad@gmail.com] On Beh= alf Of Jonathan Haddad > Sent: Friday, September 26, 2014 12:58 PM > To: user@cassandra.apache.org > Subject: Re: Repair taking long time > > If you're using DSE you might want to contact Datastax support, rather th= an the ML. > > On Fri, Sep 26, 2014 at 10:52 AM, Gene Robichaux wrote: >> I am on DSE 4.0.3 which is 2.0.7. >> >> >> >> If 4.5.1 is NOT 2.1. I guess an upgrade will not buy me much=E2=80=A6.. >> >> >> >> The bad thing is that table is not our largest=E2=80=A6.. :( >> >> >> >> >> >> Gene Robichaux >> >> Manager, Database Operations >> >> Match.com >> >> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 >> >> Phone: 214-576-3273 >> >> >> >> From: Brice Dutheil [mailto:brice.dutheil@gmail.com] >> Sent: Friday, September 26, 2014 12:47 PM >> To: user@cassandra.apache.org >> Subject: Re: Repair taking long time >> >> >> >> Unfortunately DSE 4.5.0 is still on 2.0.x >> >> >> -- Brice >> >> >> >> On Fri, Sep 26, 2014 at 7:40 PM, Jonathan Haddad wro= te: >> >> Are you using Cassandra 2.0 & vnodes? If so, repair takes forever. >> This problem is addressed in 2.1. >> >> >> On Fri, Sep 26, 2014 at 9:52 AM, Gene Robichaux >> wrote: >>> I am fairly new to Cassandra. We have a 9 node cluster, 5 in one DC >>> and 4 in another. >>> >>> >>> >>> Running a repair on a large column family seems to be moving much >>> slower than I expect. >>> >>> >>> >>> Looking at nodetool compaction stats it indicates the Validation >>> phase is running that the total bytes is 4.5T (4505336278756). >>> >>> >>> >>> This is a very large CF. The process has been running for 2.5 hours >>> and has processed 71G (71950433062). That rate is about 28.4 GB per >>> hour. At this rate it will take 158 hours, just shy of 1 week. >>> >>> >>> >>> Is this reasonable? This is my first large repair and I am wondering >>> if this is normal for a CF of this size. Seems like a long time to >>> me. >>> >>> >>> >>> Is it possible to tune this process to speed it up? Is there >>> something in my configuration that could be causing this slow >>> performance? I am running HDDs, not SSDs in a JBOD configuration. >>> >>> >>> >>> >>> >>> >>> >>> Gene Robichaux >>> >>> Manager, Database Operations >>> >>> Match.com >>> >>> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225 >>> >>> Phone: 214-576-3273 >>> >>> >> >> >> -- >> Jon Haddad >> http://www.rustyrazorblade.com >> twitter: rustyrazorblade >> >> > > > > -- > Jon Haddad > http://www.rustyrazorblade.com > twitter: rustyrazorblade --=20 Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade