From user-return-24493-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu Mar 1 17:34:05 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4F1E29D56 for ; Thu, 1 Mar 2012 17:34:05 +0000 (UTC) Received: (qmail 44447 invoked by uid 500); 1 Mar 2012 17:34:03 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 44416 invoked by uid 500); 1 Mar 2012 17:34:03 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 44408 invoked by uid 99); 1 Mar 2012 17:34:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2012 17:34:03 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a41.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2012 17:33:55 +0000 Received: from homiemail-a41.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a41.g.dreamhost.com (Postfix) with ESMTP id 3A1BF44C01C for ; Thu, 1 Mar 2012 09:33:33 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=wp9TpE9LOP VvALbupS/wE7fBikfQAjRnCcfB8/hdvvjq5AjAFAYRqOj8QSZMdiLAkAzsA26kSA ySaN/jT6Ds8nlzSpGc9ZHBaXcGGwYmgc6Nx2E6NcU62N8qupWImhdSaVB8HnCIk2 ruSYfy0Eima7Lwe/4Q7sDafDyPUKeYaIY= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=NQhK/NHcYUR7yALn OeJAOKlj/Bk=; b=jEZQ25tHjb9iPDm52R3ZIOFlYws0Sqp3JDuYIsiSre4Taa7m hmigQIPdXABki0eE9ZT+s6QF//kWkpAliU0DbvraolcDnDfoV5/c4RAd9j050IuL p4eRH9sinReEgdNmSShZ7k/7AjekGtWJd3MuucQa5u+vlovIxQ40nJ7bDG8= Received: from [172.16.1.3] (125-236-193-159.adsl.xtra.co.nz [125.236.193.159]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a41.g.dreamhost.com (Postfix) with ESMTPSA id A6B8B44C05D for ; Thu, 1 Mar 2012 09:33:32 -0800 (PST) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: multipart/alternative; boundary="Apple-Mail=_65B143C5-0F7F-4A94-A4FC-C3E86331D7BC" Subject: Re: can't find rows Date: Fri, 2 Mar 2012 06:33:28 +1300 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <2B42C3A1-DC79-4B55-91F5-AFCAB55F0F82@thelastpickle.com> X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_65B143C5-0F7F-4A94-A4FC-C3E86331D7BC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 What RF were you using and had you been running repair regularly ?=20 Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/03/2012, at 5:51 AM, Casey Deccio wrote: > On Wed, Feb 29, 2012 at 5:29 AM, Casey Deccio = wrote: > On Wed, Feb 29, 2012 at 5:25 AM, Casey Deccio = wrote: > I recently had to do some shuffling with one of my cassandra nodes = because it was running out of disk space. I did a few things in the = process, and I'm not sure in the end which caused my problem. First I = added a second file path to the data directory in cassandra.yaml. = Things still worked fine after this, as far as I could tell. Shortly = after this, however, I took down the node and rsync'd the data from both = data directories, as well as commitlogs, to an external drive. I then = shut down the machine, replaced the hard drives with bigger drives, and = re-installed the OS. I re-created the data directories, rsync'd the = data and commitlogs back over from the external drive, and started up = cassandra, re-adding it to the ring. When it came up, all of my rows = were missing for one columnfamily and nearly all my rows were missing = for another--or at least that's what it looks like, based on walking the = rows. I tried scrubbing each of the nodes. One of them had = insufficient disk space (yes, this seems to be a recurring problem) for = scrub, so I did upgradesstables instead, and that one is still in = progress. So far the scrub/upgradesstables hasn't seemed to help. But = in the log messages created during scrub/upgradesstables it shows = realistic numbers (i.e., in terms of the rows that existed before this = ordeal) created in each new sstable. Also, the loads shown when I run = nodetool ring still reflects the numbers with the complete set of rows. = That's encouraging, but I can't seem to access these phantom rows. = Please help! >=20 >=20 > I neglected to mention that I'm running cassandra 1.0.7. >=20 >=20 > Apologies for replying to my own post (again), but here's the follow = up. I decommissioned the newest of the four nodes in the cluster, which = was carrying hardly any load (I'm using ByteOrderedPartitioner), but = after I decommissioned, rows were available again, but only as they were = from 10 days ago. Supercolumns added after that date weren't around. >=20 > Casey --Apple-Mail=_65B143C5-0F7F-4A94-A4FC-C3E86331D7BC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 What = RF were you using and had you been running repair regularly = ? 

Cheers

http://www.thelastpickle.com

On 1/03/2012, at 5:51 AM, Casey Deccio wrote:

On Wed, = Feb 29, 2012 at 5:29 AM, Casey Deccio <casey@deccio.net> wrote:
On Wed, Feb 29, 2012 at 5:25 AM, Casey Deccio <casey@deccio.net> wrote:
I recently had to do some shuffling with one of my cassandra nodes = because it was running out of disk space.  I did a few things in = the process, and I'm not sure in the end which caused my problem.  = First I added a second file path to the data directory in = cassandra.yaml.  Things still worked fine after this, as far as I = could tell.  Shortly after this, however, I took down the node and = rsync'd the data from both data directories, as well as commitlogs, to = an external drive.  I then shut down the machine, replaced the hard = drives with bigger drives, and re-installed the OS.  I re-created = the data directories, rsync'd the data and commitlogs back over from the = external drive, and started up cassandra, re-adding it to the = ring.  When it came up, all of my rows were missing for one = columnfamily and nearly all my rows were missing for another--or at = least that's what it looks like, based on walking the rows.  I = tried scrubbing each of the nodes.  One of them had insufficient = disk space (yes, this seems to be a recurring problem) for scrub, so I = did upgradesstables instead, and that one is still in progress.  So = far the scrub/upgradesstables hasn't seemed to help.  But in the = log messages created during scrub/upgradesstables it shows realistic = numbers (i.e., in terms of the rows that existed before this ordeal) = created in each new sstable.  Also, the loads shown when I run = nodetool ring still reflects the numbers with the complete set of = rows.  That's encouraging, but I can't seem to access these phantom = rows.  Please help!


I neglected to mention that I'm running = cassandra 1.0.7.


Apologies for = replying to my own post (again), but here's the follow up.  I = decommissioned the newest of the four nodes in the cluster, which was = carrying hardly any load (I'm using ByteOrderedPartitioner), but after I = decommissioned, rows were available again, but only as they were from 10 = days ago.  Supercolumns added after that date weren't around.

Casey

= --Apple-Mail=_65B143C5-0F7F-4A94-A4FC-C3E86331D7BC--