cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Deccio <ca...@deccio.net>
Subject Re: can't find rows
Date Wed, 29 Feb 2012 16:51:59 GMT
On Wed, Feb 29, 2012 at 5:29 AM, Casey Deccio <casey@deccio.net> wrote:

> On Wed, Feb 29, 2012 at 5:25 AM, Casey Deccio <casey@deccio.net> wrote:
>
>> I recently had to do some shuffling with one of my cassandra nodes
>> because it was running out of disk space.  I did a few things in the
>> process, and I'm not sure in the end which caused my problem.  First I
>> added a second file path to the data directory in cassandra.yaml.  Things
>> still worked fine after this, as far as I could tell.  Shortly after this,
>> however, I took down the node and rsync'd the data from both data
>> directories, as well as commitlogs, to an external drive.  I then shut down
>> the machine, replaced the hard drives with bigger drives, and re-installed
>> the OS.  I re-created the data directories, rsync'd the data and commitlogs
>> back over from the external drive, and started up cassandra, re-adding it
>> to the ring.  When it came up, all of my rows were missing for one
>> columnfamily and nearly all my rows were missing for another--or at least
>> that's what it looks like, based on walking the rows.  I tried scrubbing
>> each of the nodes.  One of them had insufficient disk space (yes, this
>> seems to be a recurring problem) for scrub, so I did upgradesstables
>> instead, and that one is still in progress.  So far the
>> scrub/upgradesstables hasn't seemed to help.  But in the log messages
>> created during scrub/upgradesstables it shows realistic numbers (i.e., in
>> terms of the rows that existed before this ordeal) created in each new
>> sstable.  Also, the loads shown when I run nodetool ring still reflects the
>> numbers with the complete set of rows.  That's encouraging, but I can't
>> seem to access these phantom rows.  Please help!
>>
>>
> I neglected to mention that I'm running cassandra 1.0.7.
>
>
Apologies for replying to my own post (again), but here's the follow up.  I
decommissioned the newest of the four nodes in the cluster, which was
carrying hardly any load (I'm using ByteOrderedPartitioner), but after I
decommissioned, rows were available again, but only as they were from 10
days ago.  Supercolumns added after that date weren't around.

Casey

Mime
View raw message