Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 28 Sep 2012 14:07:08 +1100 (NCT)
From: "Tyler Patterson (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <1054825956.137293.1348801628817.JavaMail.jiratomcat@arcas>
In-Reply-To: <1747488827.97326.1348061467653.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (CASSANDRA-4687) Exception: DecoratedKey(xxx,
 yyy) != DecoratedKey(zzz, kkk)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465299#comment-13465299 ] 

Tyler Patterson commented on CASSANDRA-4687:
--------------------------------------------

I just spent a day trying to reproduce this error, to no avail. Here is what I tried for reference:
I wrote a python "stress" script, then ran it in 50 threads from 3 different EC2 instances, all stressing one EC2 node running cassandra-1.1.5. The processor load on the cassandra node was pegged at 100% (or very close to) the entire time each test was running.

The first version of the python stress script created text keys varying in length up to 10000 chars. It inserted and read one key at a time, where keys were randomly chosen from a pool of 10,000,000 keys. Cassandra had a key_cache size of 10mb. This test used cql, and inserted and read one key and column at a time.

The second version of the stress script worked the same as the first, but used pycassa.

Version 3 also used pycassa, but did batch_insert and multiget, each with 1000 keys at a time. It also deleted a random key after every batch_insert and multiget. For this test I also ran "cassandra-stress --operation=COUNTER_ADD" and "cassandra-stress --operation-COUNTER_GET"

version 4 was the similar to version 3, but used int32 keys instead of text.

I played with some other settings like key_cache and compaction_strategy_class during some of the tests. I never was able to get the error to happen.
                
> Exception: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk)
> -----------------------------------------------------------
>
>                 Key: CASSANDRA-4687
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4687
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.5
>         Environment: CentOS 6.3 64-bit, Oracle JRE 1.6.0.33 64-bit, single node cluster
>            Reporter: Leonid Shalupov
>            Assignee: Pavel Yaskevich
>            Priority: Critical
>             Fix For: 1.1.6
>
>         Attachments: 4687-debugging.txt
>
>
> Under heavy write load sometimes cassandra fails with assertion error.
> git bisect leads to commit 295aedb278e7a495213241b66bc46d763fd4ce66.
> works fine if global key/row caches disabled in code.
> {quote}
> java.lang.AssertionError: DecoratedKey(xxx, yyy) != DecoratedKey(zzz, kkk) in /var/lib/cassandra/data/...-he-1-Data.db
> 	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:60)
> 	at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
> 	at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
> 	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
> 	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
> 	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
> 	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
> 	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
> 	at org.apache.cassandra.db.Table.getRow(Table.java:378)
> 	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
> 	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:819)
> 	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1253)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira