Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B77D4111B8 for ; Fri, 27 Jun 2014 21:43:25 +0000 (UTC) Received: (qmail 73805 invoked by uid 500); 27 Jun 2014 21:43:25 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 73756 invoked by uid 500); 27 Jun 2014 21:43:25 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 73594 invoked by uid 99); 27 Jun 2014 21:43:25 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Jun 2014 21:43:25 +0000 Date: Fri, 27 Jun 2014 21:43:24 +0000 (UTC) From: "Jason Brown (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-7465) DecoratedKey assertion error on reads MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046430#comment-14046430 ] Jason Brown commented on CASSANDRA-7465: ---------------------------------------- similar, but perhaps completely unrelated to, CASSANDRA-7245. [~benedict] and I suspect there might be something funky with the KeyCache index > DecoratedKey assertion error on reads > ------------------------------------- > > Key: CASSANDRA-7465 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7465 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 3 nodes > Oracle Linux Server 6.3 > kernel ver 2.6.39 > Reporter: Jason Brown > Priority: Blocker > Fix For: 2.1.0 > > > Getting the following exception when running read stress: > {code}WARN [SharedPool-Worker-31] 2014-06-27 21:25:51,391 AbstractTracingAwareExecutorService.java:166 - Uncaught exception on thread Thread[SharedPool-Worker-31,10,main]: {} > java.lang.AssertionError: DecoratedKey(-5397116645141815707, 30303031393143364639) != DecoratedKey(-5397116645141815707, 30303031343439443233) in /u/sdd/cassandra-jasobrown/data/Keyspace1/Standard1-6ab9bd90fe3b11e385edff96c2ef2fd6/Keyspace1-Standard1-ka-73-Data.db > at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:111) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.db.columniterator.SSTableNamesIterator.(SSTableNamesIterator.java:59) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:89) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:125) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:59) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:341) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1393) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2060) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_13] > at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:162) ~[apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:103) [apache-cassandra-2.1.0-rc2-SNAPSHOT.jar:2.1.0-rc2-SNAPSHOT] > at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13] > {code} > I have a three node cluster, which I populate with the following stress command: > {code} > cassandra-stress write n=60000000 -schema replication\(factor\=2\) -key populate=1..60000000 -rate threads=42 -mode native prepared cql3 -port native=9043 thrift=9161 -node athena06-a,athena06-b,athena06-c -col n=fixed\(21\) size=exp\(11..42\) > {code} > Then I run the read stress: > {code} > cassandra-stress read n=100000000 -key dist=extr\(1..600000000,2\) -mode native prepared cql3 -port native=9043 thrift=9161 -node athena06-b,athena06-c,athena06-a -col n=fixed\(21\) -rate threads=64 > {code} > The above exception occurs semi-frequently (several to ~50 times a minute, but seems to depend on amount of data in cluster - anecdotal evidence only). -- This message was sent by Atlassian JIRA (v6.2#6252)