cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Boudreault (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-8366) Repair grows data on nodes, causes load to become unbalanced
Date Mon, 01 Dec 2014 18:54:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14229328#comment-14229328
] 

Alan Boudreault edited comment on CASSANDRA-8366 at 12/1/14 6:53 PM:
---------------------------------------------------------------------

I have been able to reproduce the issue with 2.1.2 and branch cassandra-2.1. From my tests,
the issue seems to be related the parallel incremental repairs. I don't see the issue with
 full repairs. With full repairs, the storage size increases but everything is fine after
a compaction.  With incremental repairs, I've seen nodes going from 1.5G to 15G of storage
size. 

It looks like something is broken with inc repairs. Most of the time, I get one of the following
errors during the repairs:

* Repair session 6f6c4ae0-78d6-11e4-9b48-b56034537865 for range (3074457345618258602,-9223372036854775808]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #6f6c4ae0-78d6-11e4-9b48-b56034537865
on r1/Standard1, (3074457345618258602,-9223372036854775808]] Sync failed between /127.0.0.1
and /127.0.0.3

* Repair failed with error Did not get positive replies from all endpoints. List of failed
endpoint(s): [127.0.0.1]

So this issue might be related to CASSANDRA-8316 .  I've attached the script I used to reproduce
the issue and also 3 result files.


was (Author: aboudreault):
I have been able to reproduce the issue with 2.1.2 and branch cassandra-2.1. From my tests,
the issue seems to be related the parallel incremental repairs. I don't see the issue with
 full repairs. With full repairs, the storage size increases but everything is fine after
a compaction.  With incremental repairs, I've seen nodes going from 1.5G to 15G of storage
size. 

It looks like something is broken with inc repairs. Most of the time, I get one of the following
errors during the repairs:

* Repair session 6f6c4ae0-78d6-11e4-9b48-b56034537865 for range (3074457345618258602,-9223372036854775808]
failed with error org.apache.cassandra.exceptions.RepairException: [repair #6f6c4ae0-78d6-11e4-9b48-b56034537865
on r1/Standard1, (3074457345618258602,-9223372036854775808]] Sync failed between /127.0.0.1
and /127.0.0.3

* Repair failed with error Did not get positive replies from all endpoints. List of failed
endpoint(s): [127.0.0.1]

So this issue might be related to CASSANDRA-8613 .  I've attached the script I used to reproduce
the issue and also 3 result files.

> Repair grows data on nodes, causes load to become unbalanced
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-8366
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8366
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: 4 node cluster
> 2.1.2 Cassandra
> Inserts and reads are done with CQL driver
>            Reporter: Jan Karlsson
>            Assignee: Alan Boudreault
>         Attachments: results-17500000_inc_repair.txt, results-5000000_1_inc_repairs.txt,
results-5000000_2_inc_repairs.txt, results-5000000_full_repair_then_inc_repairs.txt, results-5000000_inc_repairs_not_parallel.txt,
test.sh
>
>
> There seems to be something weird going on when repairing data.
> I have a program that runs 2 hours which inserts 250 random numbers and reads 250 times
per second. It creates 2 keyspaces with SimpleStrategy and RF of 3. 
> I use size-tiered compaction for my cluster. 
> After those 2 hours I run a repair and the load of all nodes goes up. If I run incremental
repair the load goes up alot more. I saw the load shoot up 8 times the original size multiple
times with incremental repair. (from 2G to 16G)
> with node 9 8 7 and 6 the repro procedure looked like this:
> (Note that running full repair first is not a requirement to reproduce.)
> After 2 hours of 250 reads + 250 writes per second:
> UN  9  583.39 MB  256     ?       28220962-26ae-4eeb-8027-99f96e377406  rack1
> UN  8  584.01 MB  256     ?       f2de6ea1-de88-4056-8fde-42f9c476a090  rack1
> UN  7  583.72 MB  256     ?       2b6b5d66-13c8-43d8-855c-290c0f3c3a0b  rack1
> UN  6  583.84 MB  256     ?       b8bd67f1-a816-46ff-b4a4-136ad5af6d4b  rack1
> Repair -pr -par on all nodes sequentially
> UN  9  746.29 MB  256     ?       28220962-26ae-4eeb-8027-99f96e377406  rack1
> UN  8  751.02 MB  256     ?       f2de6ea1-de88-4056-8fde-42f9c476a090  rack1
> UN  7  748.89 MB  256     ?       2b6b5d66-13c8-43d8-855c-290c0f3c3a0b  rack1
> UN  6  758.34 MB  256     ?       b8bd67f1-a816-46ff-b4a4-136ad5af6d4b  rack1
> repair -inc -par on all nodes sequentially
> UN  9  2.41 GB    256     ?       28220962-26ae-4eeb-8027-99f96e377406  rack1
> UN  8  2.53 GB    256     ?       f2de6ea1-de88-4056-8fde-42f9c476a090  rack1
> UN  7  2.6 GB     256     ?       2b6b5d66-13c8-43d8-855c-290c0f3c3a0b  rack1
> UN  6  2.17 GB    256     ?       b8bd67f1-a816-46ff-b4a4-136ad5af6d4b  rack1
> after rolling restart
> UN  9  1.47 GB    256     ?       28220962-26ae-4eeb-8027-99f96e377406  rack1
> UN  8  1.5 GB     256     ?       f2de6ea1-de88-4056-8fde-42f9c476a090  rack1
> UN  7  2.46 GB    256     ?       2b6b5d66-13c8-43d8-855c-290c0f3c3a0b  rack1
> UN  6  1.19 GB    256     ?       b8bd67f1-a816-46ff-b4a4-136ad5af6d4b  rack1
> compact all nodes sequentially
> UN  9  989.99 MB  256     ?       28220962-26ae-4eeb-8027-99f96e377406  rack1
> UN  8  994.75 MB  256     ?       f2de6ea1-de88-4056-8fde-42f9c476a090  rack1
> UN  7  1.46 GB    256     ?       2b6b5d66-13c8-43d8-855c-290c0f3c3a0b  rack1
> UN  6  758.82 MB  256     ?       b8bd67f1-a816-46ff-b4a4-136ad5af6d4b  rack1
> repair -inc -par on all nodes sequentially
> UN  9  1.98 GB    256     ?       28220962-26ae-4eeb-8027-99f96e377406  rack1
> UN  8  2.3 GB     256     ?       f2de6ea1-de88-4056-8fde-42f9c476a090  rack1
> UN  7  3.71 GB    256     ?       2b6b5d66-13c8-43d8-855c-290c0f3c3a0b  rack1
> UN  6  1.68 GB    256     ?       b8bd67f1-a816-46ff-b4a4-136ad5af6d4b  rack1
> restart once more
> UN  9  2 GB       256     ?       28220962-26ae-4eeb-8027-99f96e377406  rack1
> UN  8  2.05 GB    256     ?       f2de6ea1-de88-4056-8fde-42f9c476a090  rack1
> UN  7  4.1 GB     256     ?       2b6b5d66-13c8-43d8-855c-290c0f3c3a0b  rack1
> UN  6  1.68 GB    256     ?       b8bd67f1-a816-46ff-b4a4-136ad5af6d4b  rack1
> Is there something im missing or is this strange behavior?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message