cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prakash Chauhan <prakash.chau...@ericsson.com>
Subject RE: Exception in logs using LCS .
Date Thu, 30 Jun 2016 07:42:07 GMT
Hi  Paulo,

Thanks for the reply. Running scrub every time the exception occurs is not a possible solution
for us. We don’t have a log patrolling system in place that verify the log on regular basis.

I have some queries based on your reply:

1.       You mentioned race condition. Can you please elaborate a little more what actual
race condition may be happening in this scenario ?

2.       What if this happens in production ? What would be the impact? Can we live with it?

3.       Test system where the problem was reported has been scrapped so we can’t run user
defined compaction now. Any suggestions to reproduce it on a new system?



From: Paulo Motta [mailto:pauloricardomg@gmail.com]
Sent: Tuesday, June 28, 2016 5:43 PM
To: user@cassandra.apache.org
Subject: Re: Exception in logs using LCS .

1. Not necessarily data corruption, but it seems compaction is trying to write data in the
wrong order most likely due to a temporary race condition/bug a la #9935, but since the compaction
fails your original data is probably safe (you can try running scrub to verify/fix corruptions).
2. This is pretty tricky to reproduce because it will depend on which sstables were picked
for compaction at a particular instant, but you could try running a user-defined compaction
or scrub on the sstables that contain this key, see CASSANDRA-11337and https://gist.github.com/jeromatron/e238e5795b3e79866b83
3. clone the cassandra repository of your current version, git-cherry-pick the commit of CASSANDRA-9935,
ant jar, replace the cassandra jar with the generated SNAPSHOT.jar and restart the node`

2016-06-28 7:55 GMT-03:00 Prakash Chauhan <prakash.chauhan@ericsson.com<mailto:prakash.chauhan@ericsson.com>>:
Hello,

Recently we changed compaction strategy fora table to LCS from the default STCS. While bootstrapping
a node , we are getting following Exception in the logs:

ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java (line 258) Exception
in thread Thread[CompactionExecutor:81,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(-2711050270696519088, 623330333a313a35)
>= current key DecoratedKey(-8631371593982690738, 623437393a313a30) writing into /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957 ColumnFamilyStore.java (line 795) Enqueuing
flush of Memtable-compactions_in_progress@541886922(0/0 serialized/live bytes, 1 ops)

Questions:

1.       Are these exceptions due to data corruption ?

2.       I am unable to reproduce the problem. How can I reproduce the Exception? Is there
any specific case when such exceptions are raised?

3.       Without reproducing the Exception, how can I test the patch available at related
JIRA : https://issues.apache.org/jira/browse/CASSANDRA-9935



Thanks,
Prakash Chauhan.








Mime
View raw message