Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of casey@deccio.net designates
 209.85.214.172 as permitted sender)
Received-SPF: pass (google.com: domain of casey@deccio.net designates
 10.182.51.73 as permitted sender) client-ip=10.182.51.73;
MIME-Version: 1.0
In-Reply-To: 
 <CAEKtLiT1N43BVL71zvuKQk6EUifWMk5yyeWMkb5tfNypTWHe6A@mail.gmail.com>
References: 
 <CAEKtLiRW0VvAJP+WKGXF2mk9rCp8ZGdxrxtDOrZuTbRjfR1=zw@mail.gmail.com>
	<CAEKtLiT1N43BVL71zvuKQk6EUifWMk5yyeWMkb5tfNypTWHe6A@mail.gmail.com>
Date: Wed, 29 Feb 2012 08:51:59 -0800
Message-ID: 
 <CAEKtLiQ+VAwu5U+NCBzw5bPVed=f1TaGuztfObPaWCGrXx5jqw@mail.gmail.com>
Subject: Re: can't find rows
From: Casey Deccio <casey@deccio.net>
To: user <user@cassandra.apache.org>
Content-Type: multipart/alternative; boundary=f46d044470db7499c704ba1d29a4

--f46d044470db7499c704ba1d29a4
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Feb 29, 2012 at 5:29 AM, Casey Deccio <casey@deccio.net> wrote:

> On Wed, Feb 29, 2012 at 5:25 AM, Casey Deccio <casey@deccio.net> wrote:
>
>> I recently had to do some shuffling with one of my cassandra nodes
>> because it was running out of disk space.  I did a few things in the
>> process, and I'm not sure in the end which caused my problem.  First I
>> added a second file path to the data directory in cassandra.yaml.  Things
>> still worked fine after this, as far as I could tell.  Shortly after this,
>> however, I took down the node and rsync'd the data from both data
>> directories, as well as commitlogs, to an external drive.  I then shut down
>> the machine, replaced the hard drives with bigger drives, and re-installed
>> the OS.  I re-created the data directories, rsync'd the data and commitlogs
>> back over from the external drive, and started up cassandra, re-adding it
>> to the ring.  When it came up, all of my rows were missing for one
>> columnfamily and nearly all my rows were missing for another--or at least
>> that's what it looks like, based on walking the rows.  I tried scrubbing
>> each of the nodes.  One of them had insufficient disk space (yes, this
>> seems to be a recurring problem) for scrub, so I did upgradesstables
>> instead, and that one is still in progress.  So far the
>> scrub/upgradesstables hasn't seemed to help.  But in the log messages
>> created during scrub/upgradesstables it shows realistic numbers (i.e., in
>> terms of the rows that existed before this ordeal) created in each new
>> sstable.  Also, the loads shown when I run nodetool ring still reflects the
>> numbers with the complete set of rows.  That's encouraging, but I can't
>> seem to access these phantom rows.  Please help!
>>
>>
> I neglected to mention that I'm running cassandra 1.0.7.
>
>
Apologies for replying to my own post (again), but here's the follow up.  I
decommissioned the newest of the four nodes in the cluster, which was
carrying hardly any load (I'm using ByteOrderedPartitioner), but after I
decommissioned, rows were available again, but only as they were from 10
days ago.  Supercolumns added after that date weren't around.

Casey

--f46d044470db7499c704ba1d29a4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Feb 29, 2012 at 5:29 AM, Casey Deccio <span dir=3D"ltr">&lt;<a href=
=3D"mailto:casey@deccio.net" target=3D"_blank">casey@deccio.net</a>&gt;</sp=
an> wrote:<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" =
style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div>On Wed, Feb 29, 2012 at 5:25 AM, Casey Deccio <span dir=3D"ltr">&lt;<a=
 href=3D"mailto:casey@deccio.net" target=3D"_blank">casey@deccio.net</a>&gt=
;</span> wrote:<br></div><div class=3D"gmail_quote"><div>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
I recently had to do some shuffling with one of my cassandra nodes because =
it was running out of disk space.=A0 I did a few things in the process, and=
 I&#39;m not sure in the end which caused my problem.=A0 First I added a se=
cond file path to the data directory in cassandra.yaml.=A0 Things still wor=
ked fine after this, as far as I could tell.=A0 Shortly after this, however=
, I took down the node and rsync&#39;d the data from both data directories,=
 as well as commitlogs, to an external drive.=A0 I then shut down the machi=
ne, replaced the hard drives with bigger drives, and re-installed the OS.=
=A0 I re-created the data directories, rsync&#39;d the data and commitlogs =
back over from the external drive, and started up cassandra, re-adding it t=
o the ring.=A0 When it came up, all of my rows were missing for one columnf=
amily and nearly all my rows were missing for another--or at least that&#39=
;s what it looks like, based on walking the rows.=A0 I tried scrubbing each=
 of the nodes.=A0 One of them had insufficient disk space (yes, this seems =
to be a recurring problem) for scrub, so I did upgradesstables instead, and=
 that one is still in progress.=A0 So far the scrub/upgradesstables hasn=
9;t seemed to help.=A0 But in the log messages created during scrub/upgrade=
sstables it shows realistic numbers (i.e., in terms of the rows that existe=
d before this ordeal) created in each new sstable.=A0 Also, the loads shown=
 when I run nodetool ring still reflects the numbers with the complete set =
of rows.=A0 That&#39;s encouraging, but I can&#39;t seem to access these ph=
antom rows.=A0 Please help!<br>


<br></blockquote></div><div><br>I neglected to mention that I&#39;m running=
 cassandra 1.0.7.<br><br></div></div></blockquote><div><br>Apologies for re=
plying to my own post (again), but here&#39;s the follow up.=A0 I decommiss=
ioned the newest of the four nodes in the cluster, which was carrying hardl=
y any load (I&#39;m using ByteOrderedPartitioner), but after I decommission=
ed, rows were available again, but only as they were from 10 days ago.=A0 S=
upercolumns added after that date weren&#39;t around.<br>
<br>Casey<br>
</div></div>

--f46d044470db7499c704ba1d29a4--