cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitrii Saprykin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9126) java.lang.RuntimeException: Last written key DecoratedKey >= current key DecoratedKey
Date Sat, 22 Aug 2015 11:58:46 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708008#comment-14708008
] 

Dmitrii Saprykin commented on CASSANDRA-9126:
---------------------------------------------

I also experience the same issue. I did 'nodetool scrub' for affected columnfamily without
any errors. And issue persists after scrub

We use:
Cassandra 2.0.15, LCS, HSHA. 

Keyspace is 8 nodes, RF=3, ~300GB per node (~1TB of unique data without RF), ~10.000.000 keys

Before scrub we had:
{noformat}
ERROR [CompactionExecutor:499] 2015-08-21 20:26:24,438 CassandraDaemon.java (line 258) Exception
in thread Thread[CompactionExecutor:499,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1928386083417163406, 0313ad7e) >=
current key DecoratedKey(1911131194341551311, 010b2fb4) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-26880-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:498] 2015-08-21 20:29:14,655 CassandraDaemon.java (line 258) Exception
in thread Thread[CompactionExecutor:498,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1928386083417163406, 0313ad7e) >=
current key DecoratedKey(1911131194341551311, 010b2fb4) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-26954-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

After scrub I see following errors during compaction:
{noformat}
ERROR [CompactionExecutor:815] 2015-08-22 03:16:17,153 CassandraDaemon.java (line 258) Exception
in thread Thread[CompactionExecutor:815,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656, 02c3a415) >=
current key DecoratedKey(1916775706087412870, 013a445b) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-28982-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:819] 2015-08-22 03:18:10,152 CassandraDaemon.java (line 258) Exception
in thread Thread[CompactionExecutor:819,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656, 02c3a415) >=
current key DecoratedKey(1916775706087412870, 013a445b) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-28999-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:33,374 Validator.java (line 255) Failed creating
a merkle tree for [repair #a76b0d50-48a6-11e5-849b-3bfefac09ede on bigks/chunks, (1916658571049540871,1922266576662767350]],
/130.14.18.93 (see log for details)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:33,374 CassandraDaemon.java (line 258) Exception
in thread Thread[ValidationExecutor:592,1,main]
java.lang.AssertionError: row DecoratedKey(1916775706087412870, 013a445b) received out of
order wrt DecoratedKey(1922197470632862656, 02c3a415)
        at org.apache.cassandra.repair.Validator.add(Validator.java:137)
        at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
        at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
        at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:37,287 Validator.java (line 255) Failed creating
a merkle tree for [repair #aaab6b40-48a6-11e5-849b-3bfefac09ede on bigks/chunks, (1924415850871700726,1932964198982195249]],
/130.14.18.93 (see log for details)
ERROR [ValidationExecutor:592] 2015-08-22 03:20:37,287 CassandraDaemon.java (line 258) Exception
in thread Thread[ValidationExecutor:592,1,main]
java.lang.AssertionError: row DecoratedKey(1924416666703877076, 00e57790) received out of
order wrt DecoratedKey(1928782589289822441, 030cb3d9)
        at org.apache.cassandra.repair.Validator.add(Validator.java:137)
        at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:806)
        at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
        at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:404)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:1005] 2015-08-22 06:07:15,676 CassandraDaemon.java (line 258) Exception
in thread Thread[CompactionExecutor:1005,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656, 02c3a415) >=
current key DecoratedKey(1916775706087412870, 013a445b) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-29048-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
ERROR [CompactionExecutor:1009] 2015-08-22 06:11:24,670 CassandraDaemon.java (line 258) Exception
in thread Thread[CompactionExecutor:1009,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(1922197470632862656, 02c3a415) >=
current key DecoratedKey(1916775706087412870, 013a445b) writing into /data/cassandra/storage/data/bigks/chunks/bigks-chunks-tmp-jb-29071-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:170)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

> java.lang.RuntimeException: Last written key DecoratedKey >= current key DecoratedKey
> -------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-9126
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9126
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: srinivasu gottipati
>            Priority: Critical
>             Fix For: 2.0.x
>
>
> Cassandra V: 2.0.14,
> Getting the following exceptions while trying to compact (I see this issue was raised
in earlier versions and marked as closed. However it still appears in 2.0.14). In our case,
compaction is not getting succeeded and keep failing with this error.:
> {code}java.lang.RuntimeException: Last written key DecoratedKey(3462767860784856708,
354038323137333038305f3330325f31355f474d4543454f) >= current key DecoratedKey(3462334604624154281,
354036333036353334315f3336315f31355f474d4543454f) writing into {code}
> .......
> Stacktrace:{code}
> 	at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
> 	at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
> 	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> 	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
> 	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> 	at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745){code}
> Any help is greatly appreciated



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message