incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramesh Natarajan <rames...@gmail.com>
Subject Re: gracefully recover from data file corruptions
Date Sat, 17 Dec 2011 00:26:59 GMT
Thanks Ben and Jeremiah. We are actively working with our 3rd party
vendors to determine the root cause for this issue. Hopefully we will
figure something out. This repair procedure is more like a last resort
which i really don't want to use but something to keep in mind if such
necessity arises.

thanks
Ramesh

On Fri, Dec 16, 2011 at 12:48 PM, Ben Coverston
<ben.coverston@datastax.com> wrote:
> Hi Ramesh,
>
> Every time I have seen this in the last year it has been caused by bad
> hardware or bad memory. Usually we find errors in the syslog.
>
> Jeremiah is right about running repair when you get your nodes back up.
>
> Fortunately with the addition of checksums in 1.0 I don't think that the
> corrupt data can get propagated across nodes.
>
> Your recovery steps do seem solid, if not a bit verbose. I usually tell
> people to shut down the node, remove the offending SSTables, bring the node
> back up then run repair.
>
> I can't stress enough however that if you're going to bring it back up on
> the same hardware you probably want to find the root cause, otherwise you're
> going to find yourself in the same situation days/weeks/months in the
> future.
>
> Ben
>
> On Fri, Dec 16, 2011 at 5:16 PM, Jeremiah Jordan
> <jeremiah.jordan@morningstar.com> wrote:
>>
>> You need to run repair on the node once it is back up (to get back the
>> data you just deleted).  If this is happening on more than one node you
>> could have data loss...
>>
>> -Jeremiah
>>
>>
>> On 12/16/2011 07:46 AM, Ramesh Natarajan wrote:
>>>
>>> We are running a 30 node 1.0.5 cassandra cluster  running RHEL 5.6
>>> x86_64 virtualized on ESXi 5.0. We are seeing Decorated Key assertion
>>> error during compactions and at this point we are suspecting anything
>>> from OS/ESXi/HBA/iSCSI RAID.  Please correct me i am wrong, once a
>>> node gets into this state I don't see any way to recover unless I
>>> remove the corrupted data file and restart cassandra. I am running
>>> tests with replication factor 3 and all reads and writes are done with
>>> QUORUM. So i believe there will not be data loss if i do this.
>>>
>>> If this is a correct way to recover I would like to know how to
>>> gracefully do this in production environment..
>>>
>>> - Disable thrift
>>> - Disable gossip
>>> - Drain the node
>>> - kill the cassandra java process ( send a sigterm and or sigkill )
>>> - do a filesystem sync
>>> - remove the corrupted file from the /var/lib/cassandra/data directory
>>> - start cassandra
>>> - enable gossip so all pending hintedhandoff occurs
>>> - enable thrift.
>>>
>>> Thanks
>>> Ramesh
>
>
>
>
> --
> Ben Coverston
> DataStax -- The Apache Cassandra Company
>

Mime
View raw message