From user-return-37271-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Mon Oct 28 06:31:00 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0B0510932 for ; Mon, 28 Oct 2013 06:31:00 +0000 (UTC) Received: (qmail 51022 invoked by uid 500); 28 Oct 2013 06:30:56 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 50909 invoked by uid 500); 28 Oct 2013 06:30:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 50790 invoked by uid 99); 28 Oct 2013 06:30:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Oct 2013 06:30:55 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.160.47] (HELO mail-pb0-f47.google.com) (209.85.160.47) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Oct 2013 06:30:49 +0000 Received: by mail-pb0-f47.google.com with SMTP id rq2so6017901pbb.34 for ; Sun, 27 Oct 2013 23:30:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=IRq4ltJ0jPKhdncDXqKRXo/gZ0DYgCAcZkFL9CZzBko=; b=NBzqJijaA5ap1zn3h+helYXDqQ/fXLnJYxudI9CsgqjxpqkUmE73GdSNjgvGZUv14y QRoqH19p6QIf1h7tQyQMzaZJJoVj1ZN5cjcq13LCJY4aY4LC0g8gasJstrO1lYzCOu7p grHdBW6f2QY4ZvM8QOgYuSYV9ZJcIrR2QfY5+J9Tx3X1cty2Jgh+f6ueOksU8oYb+pYH kWFDsa8Nf4I8N7wVX66yylJu2GmfcBZB3FG1WCB+ZigMGpz+2m9sm3e703Ygl/6je04z R3Kj7K4PmwBaLGci7to8VYMAAyI4Lwe7yviJXvZWbwMSqNnHx5aPJ0T9C+3gJpJqMW3K u9jQ== X-Gm-Message-State: ALoCoQlmMcoFKjl07uxAZ3YzT/b5cWQD734jMUKXWCEcG1ObV6buU+4VcW+sZYFlEiRuqBSzKSsA X-Received: by 10.68.229.2 with SMTP id sm2mr14541143pbc.68.1382941827280; Sun, 27 Oct 2013 23:30:27 -0700 (PDT) Received: from [172.16.1.20] ([203.86.207.101]) by mx.google.com with ESMTPSA id ry4sm32478984pab.4.2013.10.27.23.30.25 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 27 Oct 2013 23:30:26 -0700 (PDT) From: Aaron Morton Content-Type: multipart/alternative; boundary="Apple-Mail=_D3B5F2DE-53FE-49CE-A91F-3CBDA0DB311A" Message-Id: <32A2FBE5-C841-470E-B525-249CE33D293C@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1816\)) Subject: Re: manual read repair Date: Mon, 28 Oct 2013 19:30:22 +1300 References: To: Cassandra User In-Reply-To: X-Mailer: Apple Mail (2.1816) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_D3B5F2DE-53FE-49CE-A91F-3CBDA0DB311A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > We have seen read repair take very long time even for few GBs=20 Read Repair is a process that runs during a read to repair differences = in the background. It=92s active on (by default) 10% of the reads.=20 I assume you mean nodetool repair (aka anti entropy). It runs in two = phases, first it calculates a hash of the data on the node and second it = transfers data to resolve inconsistencies.=20 You can track the first part using nodetool compactionstats and the = second with nodetool netstats.=20 I would guess it=92s the first part that is taking a while, how much CPU = power do you have ? Also the first part if throttled by the = compaction_throughput YAML setting.=20 Cheers ----------------- Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 26/10/2013, at 2:54 pm, Baskar Duraikannu = wrote: >=20 > We have seen read repair take very long time even for few GBs of data = even though we don't see disk or network bottlenecks. Do you use any = specific configuration to speed up read repairs? --Apple-Mail=_D3B5F2DE-53FE-49CE-A91F-3CBDA0DB311A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252
We have seen read repair take = very long time even for few GBs 
Read Repair is a = process that runs during a read to repair differences in the background. = It=92s active on (by default) 10% of the = reads. 

I assume you mean nodetool repair (aka = anti entropy). It runs in two phases, first it calculates a hash of the = data on the node and second it transfers data to resolve = inconsistencies. 

You can track the first = part using nodetool compactionstats and the second with nodetool = netstats. 

I would guess it=92s the first = part that is taking a while, how much CPU power do you have ? Also the = first part if throttled by the compaction_throughput YAML = setting. 

Cheers

http://www.thelastpickle.com

On 26/10/2013, at 2:54 pm, Baskar Duraikannu <baskar.duraikannu@outlook.co= m> wrote:


We = have seen read repair take very long time even for few GBs of data even = though we don't see disk or network bottlenecks. Do you use any specific = configuration to speed up read repairs? =

= --Apple-Mail=_D3B5F2DE-53FE-49CE-A91F-3CBDA0DB311A--