From user-return-33873-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu May 2 18:31:48 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8CFBE10236 for ; Thu, 2 May 2013 18:31:48 +0000 (UTC) Received: (qmail 20671 invoked by uid 500); 2 May 2013 18:31:46 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 20653 invoked by uid 500); 2 May 2013 18:31:45 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 20645 invoked by uid 99); 2 May 2013 18:31:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 May 2013 18:31:45 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a56.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 May 2013 18:31:39 +0000 Received: from homiemail-a56.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a56.g.dreamhost.com (Postfix) with ESMTP id E9972FE069 for ; Thu, 2 May 2013 11:31:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=Bg0DNqSfYDYI318in/O2BOylJP U=; b=NCZSSRllzHEq7fKoJ4QZJ+sKpbFWptsYMtycmqWmqIT77jMZZ0lzdw8znY Za1YSGqarlCdy00GAUfZzLyr9jRktUVexpEhmsN642hlJ5989iZi8nsSvQas7Jk1 yq9v0HnepKiWaFeQKXwjocgk1YF3+46IZmA6kuc2SAL7pje5Y= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a56.g.dreamhost.com (Postfix) with ESMTPSA id 38857FE065 for ; Thu, 2 May 2013 11:31:17 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_283C26ED-3524-4B0D-A0F5-328187E31ECF" Message-Id: <9621117F-50AB-4342-ABF3-CC48F4575174@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: Repair session failed Date: Fri, 3 May 2013 06:31:15 +1200 References: <84B566FB5B7B244B81E6F1FEADA9087701D8CA8916@LDNPCMMGMB01.INTRANET.BARCAPINT.COM> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1503) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_283C26ED-3524-4B0D-A0F5-328187E31ECF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hold off on running scrub (but yes it's an online operation). This is an = issue with the token ranges.=20 What version are you using ?=20 Are you using vNodes ? Can you share the output of nodetool ring (if no vnodes) or nodetool = status (if using vnodes) ? Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 2/05/2013, at 3:08 AM, Haithem Jarraya = wrote: > Can I run scrub while the node is in the ring and receiving writes? > Or I should disable thrift before? >=20 >=20 > On 1 May 2013 15:52, wrote: > Sounds like a job for =93nodetool scrub=94, which rewrites the SStable = rows in the correct order. After the scrub, nodetool repair should = succeed. >=20 > =20 >=20 > From: Haithem Jarraya [mailto:haithem.jarraya@struq.com]=20 > Sent: Wednesday, May 01, 2013 5:46 PM > To: user@cassandra.apache.org > Subject: Repair session failed >=20 > =20 >=20 > Hi,=20 >=20 > =20 >=20 > I am seeing this error message during repair, >=20 > =20 >=20 > INFO [AntiEntropyStage:1] 2013-05-01 14:30:54,300 = AntiEntropyService.java (line 764) [repair = #ed104480-b26a-11e2-af9b-05179fa66b76] mycolumnfamily is fully synced (1 = remaining column family to sync for this session) >=20 > ERROR [Thread-12725] 2013-05-01 14:30:54,304 StorageService.java (line = 2420) Repair session failed: >=20 > java.lang.IllegalArgumentException: Requested range intersects a local = range but is not fully contained in one; this would lead to imprecise = repair >=20 > at = org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropySe= rvice.java:175) >=20 > at = org.apache.cassandra.service.AntiEntropyService$RepairSession.(AntiE= ntropyService.java:621) >=20 > at = org.apache.cassandra.service.AntiEntropyService$RepairSession.(AntiE= ntropyService.java:610) >=20 > at = org.apache.cassandra.service.AntiEntropyService.submitRepairSession(AntiEn= tropyService.java:127) >=20 > at = org.apache.cassandra.service.StorageService.forceTableRepair(StorageServic= e.java:2480) >=20 > at = org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.j= ava:2416) >=20 > at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >=20 > at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) >=20 > at = java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >=20 > at java.util.concurrent.FutureTask.run(FutureTask.java:138) >=20 > at java.lang.Thread.run(Thread.java:662) >=20 > =20 >=20 > =20 >=20 > what does it mean imprecise repair? >=20 > Is it maybe because I went over the gcgrade period? >=20 > What you do if you go over that period? >=20 > Any hint will be valuable,=20 >=20 > Also I noticed when I run a repair on different node, I see a message = like this >=20 > =20 >=20 > [2013-05-01 14:30:54,305] Starting repair command #5, repairing 1120 = ranges for keyspace struqrealtime >=20 > =20 >=20 > I have couple of questions, why I have repair command #5? >=20 > And why the ranges values changes from one node to another? >=20 > =20 >=20 > =20 >=20 > Many Thanks, >=20 > =20 >=20 > H >=20 > _______________________________________________ >=20 > This message is for information purposes only, it is not a = recommendation, advice, offer or solicitation to buy or sell a product = or service nor an official confirmation of any transaction. It is = directed at persons who are professionals and is not intended for retail = customer use. Intended for recipient only. This message is subject to = the terms at: www.barclays.com/emaildisclaimer. >=20 > For important disclosures, please see: = www.barclays.com/salesandtradingdisclaimer regarding market commentary = from Barclays Sales and/or Trading, who are active market participants; = and in respect of Barclays Research, including disclosures relating to = specific issuers, please see http://publicresearch.barclays.com. >=20 > _______________________________________________ >=20 >=20 --Apple-Mail=_283C26ED-3524-4B0D-A0F5-328187E31ECF Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Hold = off on running scrub (but yes it's an online operation). This is an = issue with the token ranges. 

What version are = you using ? 
Are you using vNodes ?
Can you = share the output of nodetool ring (if no vnodes) or nodetool status (if = using vnodes) ?

Cheers

http://www.thelastpickle.com

On 2/05/2013, at 3:08 AM, Haithem Jarraya <haithem.jarraya@struq.com>= ; wrote:

Can I run scrub while the node is in the = ring and receiving writes?
Or I = should disable thrift before?


On 1 May 2013 = 15:52, <moshe.kranc@barclays.com> wrote:

Sounds like a job for =93nodetool scrub=94, which = rewrites the SStable rows in the correct order. After the scrub, = nodetool repair should succeed.

 

From: Haithem Jarraya [mailto:haithem.jarraya@struq.com]
Sent: Wednesday, May 01, 2013 5:46 PM
To: user@cassandra.apache.org
Subject: = Repair session failed

 

Hi, 

 

I am seeing this error message during = repair,

 

 INFO [AntiEntropyStage:1] 2013-05-01 = 14:30:54,300 AntiEntropyService.java (line 764) [repair = #ed104480-b26a-11e2-af9b-05179fa66b76] mycolumnfamily is fully synced (1 = remaining column family to sync for this session)

ERROR [Thread-12725] 2013-05-01 = 14:30:54,304 StorageService.java (line 2420) Repair session = failed:

java.lang.IllegalArgumentException: Requested range = intersects a local range but is not fully contained in one; this would = lead to imprecise repair

        at = org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropySe= rvice.java:175)

  =       at = org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>= (AntiEntropyService.java:621)

        at = org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>= (AntiEntropyService.java:610)

        at = org.apache.cassandra.service.AntiEntropyService.submitRepairSession(AntiEn= tropyService.java:127)

        at = org.apache.cassandra.service.StorageService.forceTableRepair(StorageServic= e.java:2480)

  =       at = org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.j= ava:2416)

        at = org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)=

      =   at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)=

        at = java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)<= u>

        = at = java.util.concurrent.FutureTask.run(FutureTask.java:138)

=

        at = java.lang.Thread.run(Thread.java:662)

 

 

what does it mean imprecise repair?

Is it maybe because I went over the gcgrade = period?

What you do = if you go over that period?

Any hint will be = valuable, 

Also = I noticed when I run a repair on different node, I see a message like = this

 

[2013-05-01 14:30:54,305] Starting repair command = #5, repairing 1120 ranges for keyspace = struqrealtime

 

I have couple = of questions, why I have repair command = #5?

And why the = ranges values changes from one node to another?

 

 

Many Thanks,

 

H

__= _____________________________________________

This message is for = information purposes only, it is not a recommendation,=20 advice, offer or solicitation to buy or sell a product or service nor an=20= official confirmation of any transaction. It is directed at persons who = are=20 professionals and is not intended for retail customer use. Intended for=20= recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer.

For = important disclosures, please see: www.barclays.com/salesandtradingdisclaimer=20 regarding market commentary from Barclays Sales and/or Trading, who are = active=20 market participants; and in respect of Barclays Research, including = disclosures=20 relating to specific issuers, please see http://publicresearch.barclays.com.

__________= _____________________________________



= --Apple-Mail=_283C26ED-3524-4B0D-A0F5-328187E31ECF--