incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Repair session failed
Date Mon, 06 May 2013 08:31:04 GMT
Can your raise a ticket at https://issues.apache.org/jira/browse/CASSANDRA  and update the
thread with the link?

Please include:
* nodetool status
* nodetool ring (so we have all the token assignments)
* The IP you started repair on 
* As much log as you can share, if you can run DEBUG for the org.apache.cassandra.service.AntiEntropyService
it would be handy. 
* the command you used to start nodetool

A range selected for the repair is not fully contained by any of ranges the node replicates.


Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 3/05/2013, at 9:02 PM, Christopher Wirt <chris.wirt@struq.com> wrote:

> Hi Aaron,
>  
> We’re running 1.2.4, so with vNodes
>  
> We ran scrub but saw the issue again when repairing
>  
> nodetool status –
>  
> Datacenter: DC01
> =================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address       Load       Tokens  Owns   Host ID                               Rack
> UN  10.70.48.23   35.16 GB   256     13.1%  4a7bc489-25af-4c20-80f8-499ffcb18e2d  RAC1
> UN  10.70.6.79    30.04 GB   256     12.6%  98a1167f-cf75-4201-a454-695e0f7d2d72  RAC1
> UN  10.70.6.78    41.94 GB   256     11.9%  62a418b5-3c38-4f66-874d-8138d6d565e5  RAC1
> UN  10.70.47.66   54.79 GB   256     13.8%  ab564d16-4081-4866-b8ba-26461d9a93d7  RAC1
> UN  10.70.6.91    46.96 GB   256     12.6%  2e1e7179-82e6-4ae6-b986-383acc9fc8a2  RAC1
> UN  10.70.47.126  38.04 GB   256     11.8%  d4bed3b1-ffaf-4c68-b560-d270355c8c4b  RAC1
> Datacenter: DC02
> =================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address       Load       Tokens  Owns   Host ID                               Rack
> UN  10.56.0.144   31.71 GB   256     12.0%  1860011e-fa7c-4ce1-ad6b-c8a38a5ddd02  RAC1
> UN  10.56.0.140   86.28 GB   256     12.3%  f3fa985d-5056-4ddc-b146-d02432c3a86e  RAC1
>  
>  
> Thanks,
>  
> Chris
>  
>  
> From: aaron morton [mailto:aaron@thelastpickle.com] 
> Sent: 02 May 2013 19:31
> To: user@cassandra.apache.org
> Subject: Re: Repair session failed
>  
> Hold off on running scrub (but yes it's an online operation). This is an issue with the
token ranges. 
>  
> What version are you using ? 
> Are you using vNodes ?
> Can you share the output of nodetool ring (if no vnodes) or nodetool status (if using
vnodes) ?
>  
> Cheers
>  
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>  
> @aaronmorton
> http://www.thelastpickle.com
>  
> On 2/05/2013, at 3:08 AM, Haithem Jarraya <haithem.jarraya@struq.com> wrote:
> 
> 
> Can I run scrub while the node is in the ring and receiving writes?
> Or I should disable thrift before?
>  
> 
> On 1 May 2013 15:52, <moshe.kranc@barclays.com> wrote:
> Sounds like a job for “nodetool scrub”, which rewrites the SStable rows in the correct
order. After the scrub, nodetool repair should succeed.
>  
> From: Haithem Jarraya [mailto:haithem.jarraya@struq.com] 
> Sent: Wednesday, May 01, 2013 5:46 PM
> To: user@cassandra.apache.org
> Subject: Repair session failed
>  
> Hi, 
>  
> I am seeing this error message during repair,
>  
>  INFO [AntiEntropyStage:1] 2013-05-01 14:30:54,300 AntiEntropyService.java (line 764)
[repair #ed104480-b26a-11e2-af9b-05179fa66b76] mycolumnfamily is fully synced (1 remaining
column family to sync for this session)
> ERROR [Thread-12725] 2013-05-01 14:30:54,304 StorageService.java (line 2420) Repair session
failed:
> java.lang.IllegalArgumentException: Requested range intersects a local range but is not
fully contained in one; this would lead to imprecise repair
>         at org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:175)
>         at org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:621)
>         at org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:610)
>         at org.apache.cassandra.service.AntiEntropyService.submitRepairSession(AntiEntropyService.java:127)
>         at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:2480)
>         at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2416)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.lang.Thread.run(Thread.java:662)
>  
>  
> what does it mean imprecise repair?
> Is it maybe because I went over the gcgrade period?
> What you do if you go over that period?
> Any hint will be valuable, 
> Also I noticed when I run a repair on different node, I see a message like this
>  
> [2013-05-01 14:30:54,305] Starting repair command #5, repairing 1120 ranges for keyspace
struqrealtime
>  
> I have couple of questions, why I have repair command #5?
> And why the ranges values changes from one node to another?
>  
>  
> Many Thanks,
>  
> H
> _______________________________________________
> 
> This message is for information purposes only, it is not a recommendation, advice, offer
or solicitation to buy or sell a product or service nor an official confirmation of any transaction.
It is directed at persons who are professionals and is not intended for retail customer use.
Intended for recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer.
> 
> For important disclosures, please see: www.barclays.com/salesandtradingdisclaimer regarding
market commentary from Barclays Sales and/or Trading, who are active market participants;
and in respect of Barclays Research, including disclosures relating to specific issuers, please
see http://publicresearch.barclays.com.
> 
> _______________________________________________
> 
>  


Mime
View raw message