Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2F276101C6 for ; Mon, 10 Jun 2013 18:12:11 +0000 (UTC) Received: (qmail 89114 invoked by uid 500); 10 Jun 2013 18:12:08 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 88913 invoked by uid 500); 10 Jun 2013 18:12:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 88905 invoked by uid 99); 10 Jun 2013 18:12:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jun 2013 18:12:06 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nimiwaribokoj@gmail.com designates 209.85.192.177 as permitted sender) Received: from [209.85.192.177] (HELO mail-pd0-f177.google.com) (209.85.192.177) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jun 2013 18:12:01 +0000 Received: by mail-pd0-f177.google.com with SMTP id p10so2197567pdj.36 for ; Mon, 10 Jun 2013 11:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:message-id:in-reply-to:references:subject :x-mailer:mime-version:content-type; bh=MCf25wBR4f7LWQqP3OiE9P5NveUKJhaHKoU1woDzFxk=; b=OYy7ZJkbJvvD15Kgs90Z869wN4TCLAbTCJXW/Ah/immQIlWmKyEe9vmiM+GgE3ZAWV tA8Lnmb3m3bLbMTAYOxNqPuq8LQ5oXCLyUfVod3WsbGUE2BKw+wisyp9+zqSmQL21z7G qsam9eU+gCGEBYbU9QynKLwyI7AdeAFukcDc3uPeHdR4tXL6Ik6KY0JC3YcnXWdVm0/7 1jM3OWrYg8jjOOQhRSRuLQoLCUGhunT5AqdjFI475Hzq7ZTRdG4RCSGDWIsYKkhHKvn4 OAQxKJhASWm33v4fdsCX8zTFe8edTL7VCOl6NnVPXunEBgp+CTL7huMDQMi+6JmnSLuZ aATQ== X-Received: by 10.68.189.36 with SMTP id gf4mr11135712pbc.73.1370887901431; Mon, 10 Jun 2013 11:11:41 -0700 (PDT) Received: from [10.0.1.33] (c-71-198-225-169.hsd1.ca.comcast.net. [71.198.225.169]) by mx.google.com with ESMTPSA id dc3sm11251542pbc.9.2013.06.10.11.11.39 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 10 Jun 2013 11:11:39 -0700 (PDT) Sender: Nimi W Date: Mon, 10 Jun 2013 11:11:37 -0700 From: Nimi Wariboko Jr To: user@cassandra.apache.org Message-ID: <6A747AD830B9440EA4669ACC8072599A@gmail.com> In-Reply-To: References: <1372322D3525455AA2CEAEC5E159FEB4@gmail.com> Subject: Re: Data Loss/Missing With Cassandra X-Mailer: sparrow 1.6.4 (build 1176) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="51b616d9_631b64d4_d01e" X-Virus-Checked: Checked by ClamAV on apache.org --51b616d9_631b64d4_d01e Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Thanks for all the advice I really appreciate it. 1.) Seeing as how I only had a single node cluster previously, if I just `nodetool move 0` the original node that should be an easy fix then? 2.) Is my data not marked as deleted? If I use sstableloader to restream the data am I just reorganizing my entire cluster or is there some logic I must use to mark deleted columns/rows as undeleted? On Monday, June 10, 2013 at 10:41 AM, Edward Capriolo wrote: > To recover it would be to dump everything then re-insert everything. > > Another option would be to return all nodes to whatever tokens they were before the switches, since the old data is still there. > > Either way both recovery options are long,painful, and a good amount of manual steps. I would not want to do either one. > > > On Mon, Jun 10, 2013 at 12:39 PM, Nimi Wariboko Jr wrote: > > How can I recover that data? Can I assume they are still in the sstables? Would doing a sstable2json then reading and reinserting be an optimal solution? > > > > > > On Monday, June 10, 2013 at 9:18 AM, Tyler Hobbs wrote: > > > > > > > > On Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr wrote: > > > > If I had to do a repair after upping the RF, than that is probably what caused the data loss. Wish I had been more careful. > > > > > > > > I'm guessing the data is irrevocably lost, I didn't make any any snapshots. > > > > > > > > Would it be possible to figure out if only a certain part of the ring was effected? That would be helpful in figuring out what data was lost. > > > > > > > > I've done a full repair now, so I'm also guessing that inconsistent data is now completely gone as well, right? > > > Cassandra doesn't remove data automatically (partially to help prevent data loss in cases like this). The original node will still have the full set of data unless you have run a cleanup operation (nodetool cleanup) on it. > > > > > > > > > -- > > > Tyler Hobbs > > > DataStax (http://datastax.com/) > > > --51b616d9_631b64d4_d01e Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Thanks for all the advice I really appreciate it. 

1.) Seeing as how I on= ly had a single node cluster previously, if I just =60nodetool move 0=60 = the original node that should be an easy fix then=3F

2.) Is my data not marked as d= eleted=3F If I use sstableloader to restream the data am I just reor= ganizing my entire cluster or is there some logic I must use to mark dele= ted columns/rows as undeleted=3F
=20

On Monday, June 10, 20= 13 at 10:41 AM, Edward Capriolo wrote:

To rec= over it would be to dump everything then re-insert everything.

Another option would be to return all nodes to whatever tokens they we= re before the switches, since the old data is still there.

Either way both recovery options are long,painful, and a good a= mount of manual steps. I would not want to do either one.
<= br>
On Mon, Jun 10, 2013 at 12:39 PM, Nimi Wariboko Jr <nimiwaribokoj=40gmail.com> wrote:
H= ow can I recover that data=3F Can I assume they are still in the sstables= =3F Would doing a sstable2json then reading and reinserting be an optimal= solution=3F

=20

On Monday, June 10, 2013= at 9:18 AM, Tyler Hobbs wrote:


On= Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr <= = nimiwaribokoj=40gmail.com> wrote:
If I had to do a repair after upping the R=46,= than that is probably what caused the data loss. Wish I had been more ca= reful.

I'm guessing the data is irrevocably lost= , I didn't make any any snapshots.

Would it be possible to figure out if only a certain = part of the ring was effected=3F That would be helpful in figuring out wh= at data was lost.

I've done a full repair now, s= o I'm also guessing that inconsistent data is now completely gone as well= , right=3F

Cassandra doesn't remove data automatically = (partially to help prevent data loss in cases like this).  The origi= nal node will still have the full set of data unless you have run a clean= up operation (nodetool cleanup) on it.


--
Tyler Hobbs
DataStax=
=20 =20 =20 =20


=20 =20 =20 =20 =20

--51b616d9_631b64d4_d01e--