cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan King <r...@twitter.com>
Subject Re: cassandra server disk full
Date Wed, 03 Aug 2011 20:11:48 GMT
The last patch on that ticket is what we're running in prod. Its
working well for us with disk_failure_mode: readwrite. In the case of
filesystem errors the node shuts off thrift and gossip. While the
gossip is propagating we can continue to serve some reads out of the
caches.

-ryan

On Tue, Aug 2, 2011 at 9:27 AM, Jim Ancona <jim@anconafamily.com> wrote:
> On Mon, Aug 1, 2011 at 6:12 PM, Ryan King <ryan@twitter.com> wrote:
>> On Fri, Jul 29, 2011 at 12:02 PM, Chris Burroughs
>> <chris.burroughs@gmail.com> wrote:
>>> On 07/25/2011 01:53 PM, Ryan King wrote:
>>>> Actually I was wrong– our patch will disable gosisp and thrift but
>>>> leave the process running:
>>>>
>>>> https://issues.apache.org/jira/browse/CASSANDRA-2118
>>>>
>>>> If people are interested in that I can make sure its up to date with
>>>> our latest version.
>>>
>>> Thanks Ryan.
>>>
>>> /me expresses interest.
>
> /me too!
>
>>>
>>> Zombie nodes when the file system does something "interesting" are not fun.
>>
>> In our experience this only gets triggered on hardware failures that
>> would otherwise seriously degrade the performance or cause lots of
>> errors.
>>
>> After the nodes traffic coalesces we get an alert which we can then deal with.
>>
>> -ryan
>>
>

Mime
View raw message