cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Rischmann ...@vrischmann.me>
Subject Re: Tools to manage repairs
Date Thu, 27 Oct 2016 15:57:08 GMT
Ok, I think we'll give incremental repairs a try on a limited number of
CFs first and then if it goes well we'll progressively switch more CFs
to incremental.

I'm not sure I understand the problem with anticompaction and
validation running concurrently. As far as I can tell, right now when a
CF is repaired (either via reaper, or via nodetool) there may be
compactions running at the same time. In fact, it happens very often.
Is it a problem ?

As far as big partitions, the biggest one we have is around 3.3Gb. Some
less big partitions are around 500Mb and less.


On Thu, Oct 27, 2016, at 05:37 PM, Alexander Dejanovski wrote:
> Oh right, that's what they advise :)
> I'd say that you should skip the full repair phase in the migration
> procedure as that will obviously fail, and just mark all sstables as
> repaired (skip 1, 2 and 6).
> Anyway you can't do better, so take a leap of faith there.
>
> Intensity is already very low and 10000 segments is a whole lot for 9
> nodes, you should not need that many.
>
> You can definitely pick which CF you'll run incremental repair on, and
> still run full repair on the rest.
> If you pick our Reaper fork, watch out for schema changes that add
> incremental repair fields, and I do not advise to run incremental
> repair without it, otherwise you might have issues with anticompaction
> and validation compactions running concurrently from time to time.
>
> One last thing : can you check if you have particularly big partitions
> in the CFs that fail to get repaired ? You can run nodetool
> cfhistograms to check that.
>
> Cheers,
>
>
>
> On Thu, Oct 27, 2016 at 5:24 PM Vincent Rischmann
> <me@vrischmann.me> wrote:
>> __
>> Thanks for the response.
>>
>> We do break up repairs between tables, we also tried our best to have
>> no overlap between repair runs. Each repair has 10000 segments
>> (purely arbitrary number, seemed to help at the time). Some runs have
>> an intensity of 0.4, some have as low as 0.05.
>>
>> Still, sometimes one particular app (which does a lot of
>> read/modify/write batches in quorum) gets slowed down to the point we
>> have to stop the repair run.
>>
>> But more annoyingly, since 2 to 3 weeks as I said, it looks like runs
>> don't progress after some time. Every time I restart reaper, it
>> starts to repair correctly again, up until it gets stuck. I have no
>> idea why that happens now, but it means I have to baby sit reaper,
>> and it's becoming annoying.
>>
>> Thanks for the suggestion about incremental repairs. It would
>> probably be a good thing but it's a little challenging to setup I
>> think. Right now running a full repair of all keyspaces (via nodetool
>> repair) is going to take a lot of time, probably like 5 days or more.
>> We were never able to run one to completion. I'm not sure it's a good
>> idea to disable autocompaction for that long.
>>
>> But maybe I'm wrong. Is it possible to use incremental repairs on
>> some column family only ?
>>
>>
>> On Thu, Oct 27, 2016, at 05:02 PM, Alexander Dejanovski wrote:
>>> Hi Vincent,
>>>
>>> most people handle repair with :
>>> - pain (by hand running nodetool commands)
>>> - cassandra range repair :
>>>   https://github.com/BrianGallew/cassandra_range_repair
>>> - Spotify Reaper
>>> - and OpsCenter repair service for DSE users
>>>
>>> Reaper is a good option I think and you should stick to it. If it
>>> cannot do the job here then no other tool will.
>>>
>>> You have several options from here :
>>>  * Try to break up your repair table by table and see which ones
>>>    actually get stuck
>>>  * Check your logs for any repair/streaming error
>>>  * Avoid repairing everything :
>>>    * you may have expendable tables
>>>    * you may have TTLed only tables with no deletes, accessed with
>>>      QUORUM CL only
>>>  * You can try to relieve repair pressure in Reaper by lowering
>>>    repair intensity (on the tables that get stuck)
>>>  * You can try adding steps to your repair process by putting a
>>>    higher segment count in reaper (on the tables that get stuck)
>>>  * And lastly, you can turn to incremental repair. As you're
>>>    familiar with Reaper already, you might want to take a look at
>>>    our Reaper fork that handles incremental repair :
>>>    https://github.com/thelastpickle/cassandra-reaper If you go down
>>>    that way, make sure you first mark all sstables as repaired
>>>    before you run your first incremental repair, otherwise you'll
>>>    end up in anticompaction hell (bad bad place) :
>>>    https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesMigration.html
>>>    Even if people say that's not necessary anymore, it'll save you
>>>    from a very bad first experience with incremental repair.
>>>    Furthermore, make sure you run repair daily after your first inc
>>>    repair run, in order to work on small sized repairs.
>>>
>>> Cheers,
>>>
>>>
>>> On Thu, Oct 27, 2016 at 4:27 PM Vincent Rischmann <me@vrischmann.me>
>>> wrote:
>>>> __
>>>> Hi,
>>>>
>>>> we have two Cassandra 2.1.15 clusters at work and are having some
>>>> trouble with repairs.
>>>>
>>>> Each cluster has 9 nodes, and the amount of data is not gigantic
>>>> but some column families have 300+Gb of data.
>>>> We tried to use `nodetool repair` for these tables but at the time
>>>> we tested it, it made the whole cluster load too much and it
>>>> impacted our production apps.
>>>>
>>>> Next we saw https://github.com/spotify/cassandra-reaper , tried it
>>>> and had some success until recently. Since 2 to 3 weeks it never
>>>> completes a repair run, deadlocking itself somehow.
>>>>
>>>> I know DSE includes a repair service but I'm wondering how do other
>>>> Cassandra users manage repairs ?
>>>>
>>>> Vincent.
>>> --
>>> -----------------
>>> Alexander Dejanovski
>>> France
>>> @alexanderdeja
>>>
>>> Consultant
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com[1]
>>
> --
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com[2]


Links:

  1. http://www.thelastpickle.com/
  2. http://www.thelastpickle.com/

Mime
View raw message