cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <>
Subject Re: can't find rows
Date Thu, 01 Mar 2012 17:33:28 GMT
What RF were you using and had you been running repair regularly ? 


Aaron Morton
Freelance Developer

On 1/03/2012, at 5:51 AM, Casey Deccio wrote:

> On Wed, Feb 29, 2012 at 5:29 AM, Casey Deccio <> wrote:
> On Wed, Feb 29, 2012 at 5:25 AM, Casey Deccio <> wrote:
> I recently had to do some shuffling with one of my cassandra nodes because it was running
out of disk space.  I did a few things in the process, and I'm not sure in the end which caused
my problem.  First I added a second file path to the data directory in cassandra.yaml.  Things
still worked fine after this, as far as I could tell.  Shortly after this, however, I took
down the node and rsync'd the data from both data directories, as well as commitlogs, to an
external drive.  I then shut down the machine, replaced the hard drives with bigger drives,
and re-installed the OS.  I re-created the data directories, rsync'd the data and commitlogs
back over from the external drive, and started up cassandra, re-adding it to the ring.  When
it came up, all of my rows were missing for one columnfamily and nearly all my rows were missing
for another--or at least that's what it looks like, based on walking the rows.  I tried scrubbing
each of the nodes.  One of them had insufficient disk space (yes, this seems to be a recurring
problem) for scrub, so I did upgradesstables instead, and that one is still in progress. 
So far the scrub/upgradesstables hasn't seemed to help.  But in the log messages created during
scrub/upgradesstables it shows realistic numbers (i.e., in terms of the rows that existed
before this ordeal) created in each new sstable.  Also, the loads shown when I run nodetool
ring still reflects the numbers with the complete set of rows.  That's encouraging, but I
can't seem to access these phantom rows.  Please help!
> I neglected to mention that I'm running cassandra 1.0.7.
> Apologies for replying to my own post (again), but here's the follow up.  I decommissioned
the newest of the four nodes in the cluster, which was carrying hardly any load (I'm using
ByteOrderedPartitioner), but after I decommissioned, rows were available again, but only as
they were from 10 days ago.  Supercolumns added after that date weren't around.
> Casey

View raw message