cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5424) nodetool repair -pr on all nodes won't repair the full range when a Keyspace isn't in all DC's
Date Tue, 16 Apr 2013 15:43:15 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632933#comment-13632933
] 

Jonathan Ellis commented on CASSANDRA-5424:
-------------------------------------------

Some questions:
- Were we relying on the Set behavior to de-duplicate entries in {{replicas}} before copying
it into an ArrayList at the end, or was that just a case of being over-cautious?
- Why don't we need to check {{ranges.size > 0}} any more in {{forceRepairAsync}}?
- Do we need to fix other uses of {{tokenMetadata.getPrimaryRangesFor}} such as {{SS.sampleKeyRange}}?
- Can we use {{getCachedEndpoints}} instead of {{calculateNaturalEndpoints}}?

Also:
- It's probably worth adding some comments to {{getPrimaryRangesForEndpoint}} -- superficially,
it looks like it is incorrect since it is still using the non-Strategy-aware {{metadata.getPredecessor}},
but after working some examples I am satisfied that it does the right thing, as it does here.

                
> nodetool repair -pr on all nodes won't repair the full range when a Keyspace isn't in
all DC's
> ----------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5424
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5424
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.1.7
>            Reporter: Jeremiah Jordan
>            Assignee: Yuki Morishita
>            Priority: Critical
>             Fix For: 1.2.5
>
>         Attachments: 5424-1.1.txt, 5424-v2-1.2.txt
>
>
> nodetool repair -pr on all nodes won't repair the full range when a Keyspace isn't in
all DC's
> Commands follow, but the TL;DR of it, range (127605887595351923798765477786913079296,0]
doesn't get repaired between .38 node and .236 node until I run a repair, no -pr, on .38
> It seems like primary arnge calculation doesn't take schema into account, but deciding
who to ask for merkle tree's from does.
> {noformat}
> Address         DC          Rack        Status State   Load            Owns         
      Token                                       
>                                                                                     
      127605887595351923798765477786913079296     
> 10.72.111.225   Cassandra   rack1       Up     Normal  455.87 KB       25.00%       
      0                                           
> 10.2.29.38      Analytics   rack1       Up     Normal  40.74 MB        25.00%       
      42535295865117307932921825928971026432      
> 10.46.113.236   Analytics   rack1       Up     Normal  20.65 MB        50.00%       
      127605887595351923798765477786913079296     
> create keyspace Keyspace1
>   with placement_strategy = 'NetworkTopologyStrategy'
>   and strategy_options = {Analytics : 2}
>   and durable_writes = true;
> -------
> # nodetool -h 10.2.29.38 repair -pr Keyspace1 Standard1
> [2013-04-03 15:46:58,000] Starting repair command #1, repairing 1 ranges for keyspace
Keyspace1
> [2013-04-03 15:47:00,881] Repair session b79b4850-9c75-11e2-0000-8b5bf6ebea9e for range
(0,42535295865117307932921825928971026432] finished
> [2013-04-03 15:47:00,881] Repair command #1 finished
> root@ip-10-2-29-38:/home/ubuntu# grep b79b4850-9c75-11e2-0000-8b5bf6ebea9e /var/log/cassandra/system.log
>  INFO [AntiEntropySessions:1] 2013-04-03 15:46:58,009 AntiEntropyService.java (line 676)
[repair #b79b4850-9c75-11e2-0000-8b5bf6ebea9e] new session: will sync a1/10.2.29.38, /10.46.113.236
on range (0,42535295865117307932921825928971026432] for Keyspace1.[Standard1]
>  INFO [AntiEntropySessions:1] 2013-04-03 15:46:58,015 AntiEntropyService.java (line 881)
[repair #b79b4850-9c75-11e2-0000-8b5bf6ebea9e] requesting merkle trees for Standard1 (to [/10.46.113.236,
a1/10.2.29.38])
>  INFO [AntiEntropyStage:1] 2013-04-03 15:47:00,202 AntiEntropyService.java (line 211)
[repair #b79b4850-9c75-11e2-0000-8b5bf6ebea9e] Received merkle tree for Standard1 from /10.46.113.236
>  INFO [AntiEntropyStage:1] 2013-04-03 15:47:00,697 AntiEntropyService.java (line 211)
[repair #b79b4850-9c75-11e2-0000-8b5bf6ebea9e] Received merkle tree for Standard1 from a1/10.2.29.38
>  INFO [AntiEntropyStage:1] 2013-04-03 15:47:00,879 AntiEntropyService.java (line 1015)
[repair #b79b4850-9c75-11e2-0000-8b5bf6ebea9e] Endpoints /10.46.113.236 and a1/10.2.29.38
are consistent for Standard1
>  INFO [AntiEntropyStage:1] 2013-04-03 15:47:00,880 AntiEntropyService.java (line 788)
[repair #b79b4850-9c75-11e2-0000-8b5bf6ebea9e] Standard1 is fully synced
>  INFO [AntiEntropySessions:1] 2013-04-03 15:47:00,880 AntiEntropyService.java (line 722)
[repair #b79b4850-9c75-11e2-0000-8b5bf6ebea9e] session completed successfully
> root@ip-10-46-113-236:/home/ubuntu# grep b79b4850-9c75-11e2-0000-8b5bf6ebea9e /var/log/cassandra/system.log
>  INFO [AntiEntropyStage:1] 2013-04-03 15:46:59,944 AntiEntropyService.java (line 244)
[repair #b79b4850-9c75-11e2-0000-8b5bf6ebea9e] Sending completed merkle tree to /10.2.29.38
for (Keyspace1,Standard1)
> root@ip-10-72-111-225:/home/ubuntu# grep b79b4850-9c75-11e2-0000-8b5bf6ebea9e /var/log/cassandra/system.log
> root@ip-10-72-111-225:/home/ubuntu# 
> -------
> # nodetool -h 10.46.113.236  repair -pr Keyspace1 Standard1
> [2013-04-03 15:48:00,274] Starting repair command #1, repairing 1 ranges for keyspace
Keyspace1
> [2013-04-03 15:48:02,032] Repair session dcb91540-9c75-11e2-0000-a839ee2ccbef for range
(42535295865117307932921825928971026432,127605887595351923798765477786913079296] finished
> [2013-04-03 15:48:02,033] Repair command #1 finished
> root@ip-10-46-113-236:/home/ubuntu# grep dcb91540-9c75-11e2-0000-a839ee2ccbef /var/log/cassandra/system.log
>  INFO [AntiEntropySessions:5] 2013-04-03 15:48:00,280 AntiEntropyService.java (line 676)
[repair #dcb91540-9c75-11e2-0000-a839ee2ccbef] new session: will sync a0/10.46.113.236, /10.2.29.38
on range (42535295865117307932921825928971026432,127605887595351923798765477786913079296]
for Keyspace1.[Standard1]
>  INFO [AntiEntropySessions:5] 2013-04-03 15:48:00,285 AntiEntropyService.java (line 881)
[repair #dcb91540-9c75-11e2-0000-a839ee2ccbef] requesting merkle trees for Standard1 (to [/10.2.29.38,
a0/10.46.113.236])
>  INFO [AntiEntropyStage:1] 2013-04-03 15:48:01,710 AntiEntropyService.java (line 211)
[repair #dcb91540-9c75-11e2-0000-a839ee2ccbef] Received merkle tree for Standard1 from a0/10.46.113.236
>  INFO [AntiEntropyStage:1] 2013-04-03 15:48:01,943 AntiEntropyService.java (line 211)
[repair #dcb91540-9c75-11e2-0000-a839ee2ccbef] Received merkle tree for Standard1 from /10.2.29.38
>  INFO [AntiEntropyStage:1] 2013-04-03 15:48:02,031 AntiEntropyService.java (line 1015)
[repair #dcb91540-9c75-11e2-0000-a839ee2ccbef] Endpoints a0/10.46.113.236 and /10.2.29.38
are consistent for Standard1
>  INFO [AntiEntropyStage:1] 2013-04-03 15:48:02,032 AntiEntropyService.java (line 788)
[repair #dcb91540-9c75-11e2-0000-a839ee2ccbef] Standard1 is fully synced
>  INFO [AntiEntropySessions:5] 2013-04-03 15:48:02,032 AntiEntropyService.java (line 722)
[repair #dcb91540-9c75-11e2-0000-a839ee2ccbef] session completed successfully
> root@ip-10-2-29-38:/home/ubuntu# grep dcb91540-9c75-11e2-0000-a839ee2ccbef /var/log/cassandra/system.log
>  INFO [AntiEntropyStage:1] 2013-04-03 15:48:01,898 AntiEntropyService.java (line 244)
[repair #dcb91540-9c75-11e2-0000-a839ee2ccbef] Sending completed merkle tree to /10.46.113.236
for (Keyspace1,Standard1)
> root@ip-10-72-111-225:/home/ubuntu# grep dcb91540-9c75-11e2-0000-a839ee2ccbef /var/log/cassandra/system.log
> root@ip-10-72-111-225:/home/ubuntu# 
> -------
> # nodetool -h 10.72.111.225  repair -pr Keyspace1 Standard1
> [2013-04-03 15:48:30,417] Starting repair command #1, repairing 1 ranges for keyspace
Keyspace1
> [2013-04-03 15:48:30,428] Repair session eeb12670-9c75-11e2-0000-316d6fba2dbf for range
(127605887595351923798765477786913079296,0] finished
> [2013-04-03 15:48:30,428] Repair command #1 finished
> root@ip-10-72-111-225:/home/ubuntu# grep eeb12670-9c75-11e2-0000-316d6fba2dbf /var/log/cassandra/system.log
>  INFO [AntiEntropySessions:1] 2013-04-03 15:48:30,427 AntiEntropyService.java (line 676)
[repair #eeb12670-9c75-11e2-0000-316d6fba2dbf] new session: will sync /10.72.111.225 on range
(127605887595351923798765477786913079296,0] for Keyspace1.[Standard1]
>  INFO [AntiEntropySessions:1] 2013-04-03 15:48:30,428 AntiEntropyService.java (line 681)
[repair #eeb12670-9c75-11e2-0000-316d6fba2dbf] No neighbors to repair with on range (127605887595351923798765477786913079296,0]:
session completed
> root@ip-10-46-113-236:/home/ubuntu# grep eeb12670-9c75-11e2-0000-316d6fba2dbf /var/log/cassandra/system.log
> root@ip-10-46-113-236:/home/ubuntu# 
> root@ip-10-2-29-38:/home/ubuntu# grep eeb12670-9c75-11e2-0000-316d6fba2dbf /var/log/cassandra/system.log
> root@ip-10-2-29-38:/home/ubuntu# 
> ---
> root@ip-10-2-29-38:/home/ubuntu# nodetool -h 10.2.29.38 repair Keyspace1 Standard1
> [2013-04-03 16:13:28,674] Starting repair command #2, repairing 3 ranges for keyspace
Keyspace1
> [2013-04-03 16:13:31,786] Repair session 6bb81c20-9c79-11e2-0000-8b5bf6ebea9e for range
(42535295865117307932921825928971026432,127605887595351923798765477786913079296] finished
> [2013-04-03 16:13:31,786] Repair session 6cb05ed0-9c79-11e2-0000-8b5bf6ebea9e for range
(0,42535295865117307932921825928971026432] finished
> [2013-04-03 16:13:31,806] Repair session 6d24a470-9c79-11e2-0000-8b5bf6ebea9e for range
(127605887595351923798765477786913079296,0] finished
> [2013-04-03 16:13:31,807] Repair command #2 finished
> root@ip-10-2-29-38:/home/ubuntu# grep 6d24a470-9c79-11e2-0000-8b5bf6ebea9e /var/log/cassandra/system.log
>  INFO [AntiEntropySessions:7] 2013-04-03 16:13:31,065 AntiEntropyService.java (line 676)
[repair #6d24a470-9c79-11e2-0000-8b5bf6ebea9e] new session: will sync a1/10.2.29.38, /10.46.113.236
on range (127605887595351923798765477786913079296,0] for Keyspace1.[Standard1]
>  INFO [AntiEntropySessions:7] 2013-04-03 16:13:31,065 AntiEntropyService.java (line 881)
[repair #6d24a470-9c79-11e2-0000-8b5bf6ebea9e] requesting merkle trees for Standard1 (to [/10.46.113.236,
a1/10.2.29.38])
>  INFO [AntiEntropyStage:1] 2013-04-03 16:13:31,751 AntiEntropyService.java (line 211)
[repair #6d24a470-9c79-11e2-0000-8b5bf6ebea9e] Received merkle tree for Standard1 from /10.46.113.236
>  INFO [AntiEntropyStage:1] 2013-04-03 16:13:31,785 AntiEntropyService.java (line 211)
[repair #6d24a470-9c79-11e2-0000-8b5bf6ebea9e] Received merkle tree for Standard1 from a1/10.2.29.38
>  INFO [AntiEntropyStage:1] 2013-04-03 16:13:31,805 AntiEntropyService.java (line 1015)
[repair #6d24a470-9c79-11e2-0000-8b5bf6ebea9e] Endpoints /10.46.113.236 and a1/10.2.29.38
are consistent for Standard1
>  INFO [AntiEntropyStage:1] 2013-04-03 16:13:31,806 AntiEntropyService.java (line 788)
[repair #6d24a470-9c79-11e2-0000-8b5bf6ebea9e] Standard1 is fully synced
>  INFO [AntiEntropySessions:7] 2013-04-03 16:13:31,806 AntiEntropyService.java (line 722)
[repair #6d24a470-9c79-11e2-0000-8b5bf6ebea9e] session completed successfully
> root@ip-10-46-113-236:/home/ubuntu# grep 6d24a470-9c79-11e2-0000-8b5bf6ebea9e /var/log/cassandra/system.log

>  INFO [AntiEntropyStage:1] 2013-04-03 16:13:31,665 AntiEntropyService.java (line 244)
[repair #6d24a470-9c79-11e2-0000-8b5bf6ebea9e] Sending completed merkle tree to /10.2.29.38
for (Keyspace1,Standard1)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message