cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francisco Nogueira Calmon Sobral <>
Subject Re: Possibly losing data with corrupted SSTables
Date Wed, 29 Jan 2014 17:59:39 GMT
Hi, Rahul.

I've run nodetool upgradesstable only in the problematic CF. It throwed the following exception:

Error occurred while upgrading the sstables for keyspace Sessions
java.util.concurrent.ExecutionException: dataSize of 3622081913630118729 starting at 32906 would be larger than
file /mnt/cassandra/data/Sessions/Users/Sessions-Users-ib-2516-Data.db length 1038
        at java.util.concurrent.FutureTask.get(
        at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(
        at org.apache.cassandra.db.compaction.CompactionManager.performSSTableRewrite(
        at org.apache.cassandra.db.ColumnFamilyStore.sstablesRewrite(
        at org.apache.cassandra.service.StorageService.upgradeSSTables(
… … 
Caused by: dataSize
of 3622081913630118729 starting at 32906 would be larger than file /mnt/cassandra/data/Sessions/Users/Sessions-Users-ib-2516-Data.db
length 1038893416
        at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(
        at org.apache.cassandra.db.compaction.CompactionManager$4.perform(
        at org.apache.cassandra.db.compaction.CompactionManager$
        ... 3 more
Caused by: dataSize of 3622081913630118729 starting at 32906 would be
larger than file /mnt/cassandra/data/Sessions/Users/Sessions-Users-ib-2516-Data.db length
        ... 20 more


On Jan 29, 2014, at 3:38 PM, Rahul Menon <> wrote:

> Francisco, 
> the sstables with *-ib-* is something that was from a previous version of c*. The *-ib-*
naming convention started at c* 1.2.1 but 1.2.10 onwards im sure it has the *-ic-* convention.
You could try running a nodetool sstableupgrade which should ideally upgrade the sstables
with the *-ib-* to *-ic-*. 
> Rahul
> On Wed, Jan 29, 2014 at 12:55 AM, Francisco Nogueira Calmon Sobral <>
> Dear experts,
> We are facing a annoying problem in our cluster.
> We have 9 amazon extra large linux nodes, running Cassandra 1.2.11.
> The short story is that after moving the data from one cluster to another, we've been
unable to run 'nodetool repair'. It get stuck due to a CorruptSSTableException in some nodes
and CFs. After looking at some problematic CFs, we observed that some of them have root permissions,
instead of cassandra permissions. Also, their names are different from the 'good' ones as
we can see below:
> ------
> -rw-r--r-- 8 cassandra cassandra 991M Nov  8 15:11 Sessions-Users-ib-2516-Data.db
> -rw-r--r-- 8 cassandra cassandra 703M Nov  8 15:11 Sessions-Users-ib-2516-Index.db
> -rw-r--r-- 8 cassandra cassandra 5.3M Nov 13 11:42 Sessions-Users-ib-2516-Summary.db
> ---------
> -rw-r--r-- 1 cassandra cassandra  22K Jan 15 10:50 Sessions-Users-ic-2933-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 106M Jan 15 10:50 Sessions-Users-ic-2933-Data.db
> -rw-r--r-- 1 cassandra cassandra 2.2M Jan 15 10:50 Sessions-Users-ic-2933-Filter.db
> -rw-r--r-- 1 cassandra cassandra  76M Jan 15 10:50 Sessions-Users-ic-2933-Index.db
> -rw-r--r-- 1 cassandra cassandra 4.3K Jan 15 10:50 Sessions-Users-ic-2933-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 574K Jan 15 10:50 Sessions-Users-ic-2933-Summary.db
> -rw-r--r-- 1 cassandra cassandra   79 Jan 15 10:50 Sessions-Users-ic-2933-TOC.txt
> We changed the permissions back to 'cassandra' and ran 'nodetool scrub' in this problematic
CF, but it has been running for at least two weeks (it is not frozen) and keeps logging many
WARNs while working with the above mentioned SSTable:
> WARN [CompactionExecutor:15] 2014-01-28 17:01:22,571 (line 57) Non-fatal
error reading row (stacktrace follows)
> Impossible row size 3618452438597849419
>         at org.apache.cassandra.db.compaction.Scrubber.scrub(
>         at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(
>         at org.apache.cassandra.db.compaction.CompactionManager.doScrub(
>         at org.apache.cassandra.db.compaction.CompactionManager.access$400(
>         at org.apache.cassandra.db.compaction.CompactionManager$3.perform(
>         at org.apache.cassandra.db.compaction.CompactionManager$
>         at
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
>         at java.util.concurrent.ThreadPoolExecutor$
>         at
> Caused by: Impossible row size 3618452438597849419
>         ... 10 more
> 1) I do not think that deleting all data of one node and running 'nodetool rebuild' will
work, since we observed that this problem occurs in all nodes. So we may not be able to restore
all the data. What can be done in this case?
> 2) Why the permissions of some sstables are 'root'? Is this problem caused by our manual
migration of data? (see long story below)
> How we ran into this?
> The long story is that we've tried to move our cluster with sstableloader, but it was
unable to load all the data correctly. Our solution was to put ALL cluster data into EACH
new node and run 'nodetool refresh'. I performed this task for each node and each column family
sequentially. Sometimes I had to rename some sstables, because they came from different nodes
with the same name. I don't remember if I ran 'nodetool repair'  or even 'nodetool cleanup'
in each node. Apparently, the process was successful, and (almost) all the data was moved.
> Unfortunately, after 3 months since we moved, I am unable to perform read operations
in some keys of some CFs. I think that some of these keys belong to the above mentioned sstables.
> Any insights are welcome.
> Best regards,
> Francisco Sobral

View raw message