Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of ryan@twitter.com designates
 209.85.218.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKYY9A+TzRggA4xnz4TW0RTPtCLnu5isjZJitxSP+3U89zm+Qg@mail.gmail.com>
References: <7DA4B654C7FEB24C9998EE4867BEE93EBC7D50@CNMAIL12.cn.utstarcom.com>
 <CALdd-zi0aF6zNwtwkt1fkLijnR0t2iXWOUancFmuNnCmKS0gng@mail.gmail.com>
 <CANpsMSExhGW5oqsQdkztpRxLasohFnwDRqWes8Cr=O2FC__+pw@mail.gmail.com>
 <CANpsMSGQvum-br55xef5hgo_JfydWcZXPy_wp++x6+cOTHA7OQ@mail.gmail.com>
 <4E3303DF.8040707@gmail.com>
 <CANpsMSEUcP323Rq=yiviBSynF+BmfrDZTmdFWPCDw6MUX4oTGQ@mail.gmail.com>
 <CAKYY9A+TzRggA4xnz4TW0RTPtCLnu5isjZJitxSP+3U89zm+Qg@mail.gmail.com>
From: Ryan King <ryan@twitter.com>
Date: Wed, 3 Aug 2011 13:11:48 -0700
Message-ID: 
 <CANpsMSG-stLQOnH9M4oFBhOM2h8H1XVKvC6eraF0pizUSoq7tw@mail.gmail.com>
Subject: Re: cassandra server disk full
To: user@cassandra.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

The last patch on that ticket is what we're running in prod. Its
working well for us with disk_failure_mode: readwrite. In the case of
filesystem errors the node shuts off thrift and gossip. While the
gossip is propagating we can continue to serve some reads out of the
caches.

-ryan

On Tue, Aug 2, 2011 at 9:27 AM, Jim Ancona <jim@anconafamily.com> wrote:
> On Mon, Aug 1, 2011 at 6:12 PM, Ryan King <ryan@twitter.com> wrote:
>> On Fri, Jul 29, 2011 at 12:02 PM, Chris Burroughs
>> <chris.burroughs@gmail.com> wrote:
>>> On 07/25/2011 01:53 PM, Ryan King wrote:
>>>> Actually I was wrong=E2=80=93 our patch will disable gosisp and thrift=
 but
>>>> leave the process running:
>>>>
>>>> https://issues.apache.org/jira/browse/CASSANDRA-2118
>>>>
>>>> If people are interested in that I can make sure its up to date with
>>>> our latest version.
>>>
>>> Thanks Ryan.
>>>
>>> /me expresses interest.
>
> /me too!
>
>>>
>>> Zombie nodes when the file system does something "interesting" are not =
fun.
>>
>> In our experience this only gets triggered on hardware failures that
>> would otherwise seriously degrade the performance or cause lots of
>> errors.
>>
>> After the nodes traffic coalesces we get an alert which we can then deal=
 with.
>>
>> -ryan
>>
>