incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei Zhu <wz1...@yahoo.com>
Subject Re: Repair Hanging C* 1.2.4
Date Thu, 02 May 2013 21:47:39 GMT
It would be helpful to have something like "nodetool repairstatus" which reports back the status
of existing repair session and recently completed repair session. 
Right now, I have to grep the log, check compactionstatus, tpstatus, netstatus to find out
what's going on with the repair. 

-Wei 

----- Original Message -----

From: "Haithem Jarraya" <haithem.jarraya@struq.com> 
To: user@cassandra.apache.org 
Sent: Thursday, May 2, 2013 6:39:37 AM 
Subject: Re: Repair Hanging C* 1.2.4 


+1 for moshe, 
I found that a cluster in a strange state due to just one node failing, defeat the USPs of
C* such as fault tolerance, data replication between DC... 





On 2 May 2013 14:27, < moshe.kranc@barclays.com > wrote: 





Nothing kills a C* ring like an unstable node that won’t die but drives the other nodes
crazy. 

Here’s an idea for a new feature: 
The Living Will: A configurable set of error conditions (e.g., disk full, heap space filling
up), which, if they happen, make the node voluntarily shut down. 


From: Haithem Jarraya [mailto: haithem.jarraya@struq.com ] 
Sent: Thursday, May 02, 2013 4:13 PM 
To: user@cassandra.apache.org 
Subject: Re: Repair Hanging C* 1.2.4 


Hi Yuki, 



I think I found what went wrong, one box in WDC had the disk full during scrub. The box become
unusable but kept alive status. 

It was not supposed to go down or marked as dead? 

By the way can we specify the snpashot output directory in nodetool command, so that does
not happen again? 



Thanks, 



H 



On 2 May 2013 13:17, Yuki Morishita < mor.yuki@gmail.com > wrote: 

Hi, 


<blockquote>



ERROR [Thread-12725] 2013-05-01 14:30:54,304 StorageService.java (line 2420) Repair session
failed: 
java.lang.IllegalArgumentException: Requested range intersects a local range but is not fully
contained in one; this would lead to imprecise repair 




This error means you are repairing the range that spreads across multiple (virtual) nodes.


I think this won't happen unless you specify the repair range with -st and -et option. 



How do you start repair? 



-- 
Yuki Morishita 
Sent with Airmail 


On May 2, 2013 at May 2, 2013, Haithem Jarraya ( haithem.jarraya@struq.com ) wrote: 
<blockquote>



Hi All, 

Cassandra repair has been a real pain for us and it’s holding back our migration from mongo
for quiet sometimes now. 
We saw errors like this during the repair, 
INFO [AntiEntropyStage:1] 2013-05-01 14:30:54,300 AntiEntropyService.java (line 764) [repair
#ed104480-b26a-11e2-af9b-05179fa66b76] mycolumnfamily is fully synced (1 remaining column
family to sync for this session) 
ERROR [Thread-12725] 2013-05-01 14:30:54,304 StorageService.java (line 2420) Repair session
failed: 
java.lang.IllegalArgumentException: Requested range intersects a local range but is not fully
contained in one; this would lead to imprecise repair 
at org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:175)

at org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:621)

at org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:610)

at org.apache.cassandra.service.AntiEntropyService.submitRepairSession(AntiEntropyService.java:127)

at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:2480)

at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2416) 
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) 
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) 
at java.util.concurrent.FutureTask.run(FutureTask.java:138) 
at java.lang.Thread.run(Thread.java:662) 


Ok we might have gone beyond the GCGrades(again before repair do not complete) So we ran scrub
in all node in parallel as it was suggested in this mailing list. 
I am not sure if this can be the cause of the problem or not, but in reality we had this issue
of repair not completing and hanging from the day we started testing cassandra 1.2.2, same
issue happening with every upgrade 1.2.3 and now 1.2.4. 
I want a way to kick in the repair if they hang or cancel the previous one without restarting
the cluster, we can’t afford to do that the day we go live. 

Let me start by presenting our current configuration. 
Data Centers: 
2 Data center (Amsterdam 6 nodes with RF of 3, Washington D.C with RF of 1) 
1 Key space with 3 column families ~= 100GB of data. 
Each node running Cassandra 1.2.4 with Java6_update45 running centos 2.6 with 32GB of RAM,
24 Cores @2.00GHZ, JNA v3.2.4 installed, 2 disk( 1 rotational for os and commit logs, and
1 ssd for the data). We are getting really good read performances, 99% < 10ms, 95% <
5ms. 

nodetool status 
Datacenter: ams01 
================= 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address Load Tokens Owns Host ID Rack 
UN x.x.x.23 34.04 GB 256 13.1% 4a7bc489-25af-4c20-80f8-499ffcb18e2d RAC1 
UN x.x.x.79 28.53 GB 256 12.6% 98a1167f-cf75-4201-a454-695e0f7d2d72 RAC1 
UN x.x.x.78 41.31 GB 256 11.9% 62a418b5-3c38-4f66-874d-8138d6d565e5 RAC1 
UN x.x.x.66 54.41 GB 256 13.8% ab564d16-4081-4866-b8ba-26461d9a93d7 RAC1 
UN x.x.x.91 45.92 GB 256 12.6% 2e1e7179-82e6-4ae6-b986-383acc9fc8a2 RAC1 
UN x.x.x.126 37.31 GB 256 11.8% d4bed3b1-ffaf-4c68-b560-d270355c8c4b RAC1 
Datacenter: wdc01 
================= 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address Load Tokens Owns Host ID Rack 
UN x.x.x.144 30.64 GB 256 12.0% 1860011e-fa7c-4ce1-ad6b-c8a38a5ddd02 RAC1 
UN x.x.x.140 86.05 GB 256 12.3% f3fa985d-5056-4ddc-b146-d02432c3a86e RAC1 

nodetool status mykeyspace 
Datacenter: ams01 
================= 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address Load Tokens Owns (effective) Host ID Rack 
UN x.x.x.66 54.41 GB 256 53.6% ab564d16-4081-4866-b8ba-26461d9a93d7 RAC1 
UN x.x.x.91 45.92 GB 256 52.1% 2e1e7179-82e6-4ae6-b986-383acc9fc8a2 RAC1 
UN x.x.x.126 37.31 GB 256 47.9% d4bed3b1-ffaf-4c68-b560-d270355c8c4b RAC1 
UN x.x.x.23 34.04 GB 256 50.9% 4a7bc489-25af-4c20-80f8-499ffcb18e2d RAC1 
UN x.x.x.79 28.53 GB 256 47.4% 98a1167f-cf75-4201-a454-695e0f7d2d72 RAC1 
UN x.x.x.78 41.31 GB 256 48.0% 62a418b5-3c38-4f66-874d-8138d6d565e5 RAC1 
Datacenter: wdc01 
================= 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address Load Tokens Owns (effective) Host ID Rack 
UN x.x.x140 86.05 GB 256 51.5% f3fa985d-5056-4ddc-b146-d02432c3a86e RAC1 
UN x.x.x.144 30.64 GB 256 48.5% 1860011e-fa7c-4ce1-ad6b-c8a38a5ddd02 RAC1 

The first thing we notice is that the data distribution is off by few %, well I guess, if
we had a repair running with no hanging this should fix the problem. 
Now when we run a repair it usually hangs after a day, there is no more repair messages in
the logs or stream, cpu usage goes down. 
This the output for nodetool tpstats and nodetool netstats in all boxes. 

Amsterdam data center: 
NODE 1: 
x.x.x.23 (I run the first repair on this node), we can see the AntiEntropySession 4 Active
and 7 pending 
nodetool tpstats 
Pool Name Active Pending Completed Blocked All time blocked 
ReadStage 0 1 33134051 0 0 
RequestResponseStage 0 0 60655547 0 0 
MutationStage 0 0 46000521 0 0 
ReadRepairStage 0 0 2712610 0 0 
ReplicateOnWriteStage 0 0 0 0 0 
GossipStage 0 0 848258 0 0 
AntiEntropyStage 0 0 24530 0 0 
MigrationStage 0 0 56 0 0 
MemtablePostFlusher 0 0 3295 0 0 
FlushWriter 0 0 1487 0 0 
MiscStage 0 0 245 0 0 
commitlog_archiver 0 0 0 0 0 
AntiEntropySessions 4 7 891 0 0 
InternalResponseStage 0 0 7 0 0 
HintedHandoff 0 0 7 0 0 

Message type Dropped 
RANGE_SLICE 0 
READ_REPAIR 0 
BINARY 0 
READ 0 
MUTATION 0 
_TRACE 0 
REQUEST_RESPONSE 0 

For nodetool stats, it says nothing streaming from x.x.x.140 which is the ip of 1 nodes in
WDC datacentre. 
nodetool netstats 
Mode: NORMAL 
Not sending any streams. 
Nothing streaming from /x.x.x.140 
Pool Name Active Pending Completed 
Commands n/a 0 60710552 
Responses n/a 0 56269681 

NODE 2: 
x.x.x.126 nothing going on this node. 
nodetool tpstats 
Pool Name Active Pending Completed Blocked All time blocked 
ReadStage 0 0 34896655 0 0 
RequestResponseStage 0 0 74815324 0 0 
MutationStage 0 0 44827842 0 0 
ReadRepairStage 0 0 3175404 0 0 
ReplicateOnWriteStage 0 0 0 0 0 
GossipStage 0 0 971226 0 0 
AntiEntropyStage 0 0 11904 0 0 
MigrationStage 0 0 59 0 0 
MemtablePostFlusher 0 0 7029 0 0 
FlushWriter 0 0 3000 0 0 
MiscStage 0 0 1300 0 0 
commitlog_archiver 0 0 0 0 0 
AntiEntropySessions 0 0 0 0 0 
InternalResponseStage 0 0 7 0 0 
HintedHandoff 0 0 8 0 0 

Message type Dropped 
RANGE_SLICE 0 
READ_REPAIR 0 
BINARY 0 
READ 0 
MUTATION 0 
_TRACE 0 
REQUEST_RESPONSE 0 

Same output for netstats as the previous node 
nodetool netstats 
Mode: NORMAL 
Not sending any streams. 
Nothing streaming from /x.x.x.140 
Pool Name Active Pending Completed 
Commands n/a 0 75174907 
Responses n/a 0 55239904 

NODE 3: 
x.x.x.78 after the repair stopped in NODE 1, I tried to run a repair on this node to see if,
it will change things or not (I think I tried to run it twice) 
We can see 4 AntiEntropySession Active and 6 Pending 
nodetool tpstats 
Pool Name Active Pending Completed Blocked All time blocked 
ReadStage 0 0 29928646 0 0 
RequestResponseStage 0 0 81431526 0 0 
MutationStage 0 0 46631197 0 0 
ReadRepairStage 0 0 3352193 0 0 
ReplicateOnWriteStage 0 0 0 0 0 
GossipStage 0 0 994857 0 0 
AntiEntropyStage 0 0 32477 0 0 
MigrationStage 0 0 44 0 0 
MemtablePostFlusher 0 0 7521 0 0 
FlushWriter 0 0 3110 0 0 
MiscStage 0 0 1023 0 0 
commitlog_archiver 0 0 0 0 0 
AntiEntropySessions 4 6 360 0 0 
InternalResponseStage 0 0 70 0 0 
HintedHandoff 0 0 9 0 0 

Message type Dropped 
RANGE_SLICE 0 
READ_REPAIR 0 
BINARY 0 
READ 0 
MUTATION 0 
_TRACE 0 
REQUEST_RESPONSE 0 


We can see for the netstats that it is waiting 
nodetool netstats 
Mode: NORMAL 
Not sending any streams. 
Nothing streaming from /x.x.x.140 
Pool Name Active Pending Completed 
Commands n/a 0 81478294 
Responses n/a 3 50728352 

NODE 4: 
x.x.x. 66 
nodetool tpstats 
Pool Name Active Pending Completed Blocked All time blocked 
ReadStage 0 0 31542526 0 0 
RequestResponseStage 0 0 66173136 0 0 
MutationStage 0 0 46796311 0 0 
ReadRepairStage 0 0 2542891 0 0 
ReplicateOnWriteStage 0 0 0 0 0 
GossipStage 0 0 726267 0 0 
AntiEntropyStage 0 0 3782 0 0 
MigrationStage 0 0 50 0 0 
MemtablePostFlusher 0 0 2807 0 0 
FlushWriter 0 0 1400 0 2 
MiscStage 0 0 679 0 0 
commitlog_archiver 0 0 0 0 0 
AntiEntropySessions 0 0 0 0 0 
InternalResponseStage 0 0 3 0 0 
HintedHandoff 0 0 8 0 0 

Message type Dropped 
RANGE_SLICE 0 
READ_REPAIR 0 
BINARY 0 
READ 0 
MUTATION 0 
_TRACE 0 
REQUEST_RESPONSE 0 

nodetool netstats 
Mode: NORMAL 
Not sending any streams. 
Nothing streaming from /x.x.x.140 
Pool Name Active Pending Completed 
Commands n/a 0 66201849 
Responses n/a 0 54649566 

NODE 5 
x.x.x.79 
nodetool tpstats 
Pool Name Active Pending Completed Blocked All time blocked 
ReadStage 0 0 4807546 0 0 
RequestResponseStage 0 0 15208415 0 0 
MutationStage 0 0 17640854 0 0 
ReadRepairStage 0 0 208035 0 0 
ReplicateOnWriteStage 0 0 0 0 0 
GossipStage 0 0 241855 0 0 
AntiEntropyStage 0 0 1096 0 0 
MigrationStage 0 0 31 0 0 
MemtablePostFlusher 0 0 801 0 0 
FlushWriter 0 0 351 0 0 
MiscStage 0 0 101 0 0 
commitlog_archiver 0 0 0 0 0 
AntiEntropySessions 0 0 0 0 0 
InternalResponseStage 0 0 10 0 0 
HintedHandoff 0 0 0 0 0 

Message type Dropped 
RANGE_SLICE 0 
READ_REPAIR 0 
BINARY 0 
READ 0 
MUTATION 0 
_TRACE 0 
REQUEST_RESPONSE 0 

# nodetool netstats 
Mode: NORMAL 
Not sending any streams. 
Nothing streaming from /x.x.x.140 
Pool Name Active Pending Completed 
Commands n/a 0 15216079 
Responses n/a 2 20432998 

NODE 6: 
x.x.x.91 
nodetool tpstats 
tPool Name Active Pending Completed Blocked All time blocked 
ReadStage 0 0 50507669 0 0 
RequestResponseStage 0 0 72430667 0 0 
MutationStage 0 0 47096834 0 0 
ReadRepairStage 0 0 3135286 0 0 
ReplicateOnWriteStage 0 0 0 0 0 
GossipStage 0 0 728625 0 0 
AntiEntropyStage 0 0 3996 0 0 
MigrationStage 0 0 57 0 0 
MemtablePostFlusher 0 0 2941 0 0 
FlushWriter 0 0 1453 0 2 
MiscStage 0 0 743 0 0 
commitlog_archiver 0 0 0 0 0 
AntiEntropySessions 0 0 0 0 0 
InternalResponseStage 0 0 3 0 0 
HintedHandoff 0 0 10 0 0 

Message type Dropped 
RANGE_SLICE 0 
READ_REPAIR 0 
BINARY 0 
READ 0 
MUTATION 0 
_TRACE 0 
REQUEST_RESPONSE 0 

nodetool netstats 
Mode: NORMAL 
Not sending any streams. 
Nothing streaming from /x.x.x.140 
Pool Name Active Pending Completed 
Commands n/a 0 73084789 
Responses n/a 0 66121629 


WDC datacentre: 
In this node we see a different tpstats, it shows one MiscStage 1 Active and 28 pending, why
is that? 
The netstats shows that all the streaming are not progressing. 
Node 1: 
x.x.x.140 
nodetool tpstats 
Pool Name Active Pending Completed Blocked All time blocked 
ReadStage 0 0 7780233 0 0 
RequestResponseStage 0 0 0 0 0 
MutationStage 0 0 36732109 0 0 
ReadRepairStage 0 0 0 0 0 
ReplicateOnWriteStage 0 0 0 0 0 
GossipStage 0 0 641497 0 0 
AntiEntropyStage 0 0 4834 0 0 
MigrationStage 0 0 65 0 0 
MemtablePostFlusher 1 5 4400 0 0 
FlushWriter 0 0 2771 0 0 
MiscStage 1 28 1720 0 0 
commitlog_archiver 0 0 0 0 0 
AntiEntropySessions 0 0 0 0 0 
InternalResponseStage 0 0 15 0 0 
HintedHandoff 0 0 8 0 0 

Message type Dropped 
RANGE_SLICE 0 
READ_REPAIR 0 
BINARY 0 
READ 0 
MUTATION 0 
_TRACE 0 
REQUEST_RESPONSE 0 

nodetool netstats 
Mode: NORMAL 
Nothing streaming to /x.x.x.91 
Streaming from: /x.x.x.91 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-31657-Data.db
sections=1192 progress=0/2001599 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-33861-Data.db
sections=8 progress=0/36381 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-33900-Data.db
sections=2 progress=0/35827 - 0% 
Streaming from: /x.x.x.23 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-20361-Data.db
sections=1 progress=0/35973 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-19809-Data.db
sections=5809 progress=0/7701015 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-20297-Data.db
sections=8 progress=0/36494 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-19444-Data.db
sections=1191 progress=0/1964863 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily3/mykeyspace-mycloumnfamily3-ib-10019-Data.db
sections=2338 progress=0/5357560 - 0% 
Streaming from: /x.x.x.78 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily3/mykeyspace-mycloumnfamily3-ib-15329-Data.db
sections=2338 progress=0/5358677 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-31112-Data.db
sections=1 progress=0/36877 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-28005-Data.db
sections=4026 progress=0/7804220 - 0% 
Streaming from: /x.x.x.66 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily2/mykeyspace-mycloumnfamily2-ib-42913-Data.db
sections=4026 progress=0/7803966 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-39649-Data.db
sections=1345 progress=0/184273 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-41289-Data.db
sections=1138 progress=0/1471186 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-42101-Data.db
sections=133 progress=0/74800 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-42399-Data.db
sections=23 progress=0/36965 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-42447-Data.db
sections=3 progress=0/36404 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-41119-Data.db
sections=2057 progress=0/1234797 - 0% 
mykeyspace: /disk1/cassandra/data/mykeyspace/mycloumnfamily1/mykeyspace-mycloumnfamily1-ib-40226-Data.db
sections=4632 progress=0/5706080 - 0% 
Pool Name Active Pending Completed 
Commands n/a 0 2826 
Responses n/a 4 40771253 

NODE 2: 
x.x.x.140 
nodetool tpstats 
Pool Name Active Pending Completed Blocked All time blocked 
ReadStage 0 0 3412912 0 0 
RequestResponseStage 0 0 4 0 0 
MutationStage 0 0 22540846 0 0 
ReadRepairStage 0 0 0 0 0 
ReplicateOnWriteStage 0 0 0 0 0 
GossipStage 0 0 296181 0 0 
AntiEntropyStage 0 0 6276 0 0 
MigrationStage 0 0 24 0 0 
MemtablePostFlusher 0 0 5092 0 0 
FlushWriter 0 0 2953 0 0 
MiscStage 0 0 1811 0 0 
commitlog_archiver 0 0 0 0 0 
AntiEntropySessions 0 0 0 0 0 
InternalResponseStage 0 0 8 0 0 
HintedHandoff 0 0 7 0 0 

Message type Dropped 
RANGE_SLICE 0 
READ_REPAIR 8942 
BINARY 0 
READ 0 
MUTATION 114559 
_TRACE 0 
REQUEST_RESPONSE 0 

nodetool netstats 
Mode: NORMAL 
Not sending any streams. 
Not receiving any streams. 
Pool Name Active Pending Completed 
Commands n/a 0 1581 
Responses n/a 0 25108161 



Many thanks, 


Haithem 
</blockquote>


_______________________________________________ 
This message is for information purposes only, it is not a recommendation, advice, offer or
solicitation to buy or sell a product or service nor an official confirmation of any transaction.
It is directed at persons who are professionals and is not intended for retail customer use.
Intended for recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer
. 
For important disclosures, please see: www.barclays.com/salesandtradingdisclaimer regarding
market commentary from Barclays Sales and/or Trading, who are active market participants;
and in respect of Barclays Research, including disclosures relating to specific issuers, please
see http://publicresearch.barclays.com . 
_______________________________________________ 
</blockquote>



Mime
View raw message