cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin <co...@clark.ws>
Subject Re: Repair/Compaction Completion Confirmation
Date Tue, 28 Oct 2014 09:53:42 GMT
When I use virtual nodes, I typically use a much smaller number - usually in the range of 10.
 This gives me the ability to add nodes easier without the performance hit.



--
Colin Clark 
+1-320-221-9531
 

> On Oct 28, 2014, at 10:46 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
> 
> I have been trying this yesterday too.
> 
> https://github.com/BrianGallew/cassandra_range_repair
> 
> "Not 100% bullet proof" --> Indeed I found that operations are done multiple times,
so it is not very optimised. Though it is open sourced so I guess you can improve things as
much as you want and contribute. Here is the issue I raised yesterday https://github.com/BrianGallew/cassandra_range_repair/issues/14.
> 
> I am also trying to improve our repair automation since we now have multiple DC and up
to 800 GB per node. Repairs are quite heavy right now.
> 
> Good luck,
> 
> Alain
> 
> 2014-10-28 4:59 GMT+01:00 Ben Bromhead <ben@instaclustr.com>:
>> https://github.com/BrianGallew/cassandra_range_repair
>> 
>> This breaks down the repair operation into very small portions of the ring as a way
to try and work around the current fragile nature of repair. 
>> 
>> Leveraging range repair should go some way towards automating repair (this is how
the automatic repair service in DataStax opscenter works, this is how we perform repairs).
>> 
>> We have had a lot of success running repairs in a similar manner against vnode enabled
clusters. Not 100% bullet proof, but way better than nodetool repair 
>> 
>> 
>> 
>>> On 28 October 2014 08:32, Tim Heckman <tim@pagerduty.com> wrote:
>>>> On Mon, Oct 27, 2014 at 1:44 PM, Robert Coli <rcoli@eventbrite.com>
wrote:
>>>>> On Mon, Oct 27, 2014 at 1:33 PM, Tim Heckman <tim@pagerduty.com>
wrote:
>>>>> I know that when issuing some operations via nodetool, the command blocks
until the operation is finished. However, is there a way to reliably determine whether or
not the operation has finished without monitoring that invocation of nodetool?
>>>>> 
>>>>> In other words, when I run 'nodetool repair' what is the best way to
reliably determine that the repair is finished without running something equivalent to a 'pgrep'
against the command I invoked? I am curious about trying to do the same for major compactions
too.
>>>> 
>>>> This is beyond a FAQ at this point, unfortunately; non-incremental repair
is awkward to deal with and probably impossible to automate. 
>>>> 
>>>> In The Future [1] the correct solution will be to use incremental repair,
which mitigates but does not solve this challenge entirely.
>>>> 
>>>> As brief meta commentary, it would have been nice if the project had spent
more time optimizing the operability of the critically important thing you must do once a
week [2].
>>>> 
>>>> https://issues.apache.org/jira/browse/CASSANDRA-5483
>>>> 
>>>> =Rob
>>>> [1] http://www.datastax.com/dev/blog/anticompaction-in-cassandra-2-1
>>>> [2] Or, more sensibly, once a month with gc_grace_seconds set to 34 days.
>>> 
>>> Thank you for getting back to me so quickly. Not the answer that I was secretly
hoping for, but it is nice to have confirmation. :)
>>> 
>>> Cheers!
>>> -Tim 
>> 
>> 
>> 
>> -- 
>> Ben Bromhead
>> 
>> Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359
> 

Mime
View raw message