Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 990BA1028E for ; Mon, 22 Jul 2013 07:12:31 +0000 (UTC) Received: (qmail 98759 invoked by uid 500); 22 Jul 2013 07:12:29 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 98744 invoked by uid 500); 22 Jul 2013 07:12:27 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 98731 invoked by uid 99); 22 Jul 2013 07:12:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jul 2013 07:12:26 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of sylvain@datastax.com designates 209.85.192.170 as permitted sender) Received: from [209.85.192.170] (HELO mail-pd0-f170.google.com) (209.85.192.170) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jul 2013 07:12:20 +0000 Received: by mail-pd0-f170.google.com with SMTP id x11so6513331pdj.29 for ; Mon, 22 Jul 2013 00:11:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=FYfcKRq4DqaZ4saLrZMK3Sz6xxDHnupvZt2J6mF1WKY=; b=WKmCN73olMmWIQD0hOBJ101ndvipNtxksi/JVRwVoQQKos/GI2aGlZjEFZZC0w8U9D hPtM4ZqMyEihAi06c9FPlFjjerbGImsFEgLppyfTwpsX6gVsAIl8f+DzMVjnDrzlI4Pv EJ3Tlvo5SJZ73Mq9NHVzu4BRknssD4djxmUa/sFb0onlIO+3QKoMf3y/paFx3M4laqi5 E/2dpMDmI4T7TcfOzVQueG243uRf+7FbElKKCJs7G/1S0CNUSPHjv56DNFuI+jpaTreI ikFv4B2wl2QSRCH1nINtR4MebzLTxsovqXbCyytyDh6S1wW195m/h2pVnofbVo371vYe oILQ== MIME-Version: 1.0 X-Received: by 10.68.212.106 with SMTP id nj10mr29122486pbc.74.1374477118290; Mon, 22 Jul 2013 00:11:58 -0700 (PDT) Received: by 10.68.88.37 with HTTP; Mon, 22 Jul 2013 00:11:58 -0700 (PDT) In-Reply-To: References: Date: Mon, 22 Jul 2013 09:11:58 +0200 Message-ID: Subject: Re: Cassandra 2.0 - AssertionError in ArrayBackedSortedColumns From: Sylvain Lebresne To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=e89a8ff1ca5263558904e2146463 X-Gm-Message-State: ALoCoQln0hTjtYAEqntjEFWQ0ygVpLei4CmPYgVsIDOpoYGl2LhrXQ0qWQPrsJZcIL0Kf9jmFWy6 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8ff1ca5263558904e2146463 Content-Type: text/plain; charset=ISO-8859-1 This is a bug really: https://issues.apache.org/jira/browse/CASSANDRA-5786. This should get fixed in the next beta of 2.0, but if you really want to test CAS updates in the meantime, you'll have to provide the columns in (column family comparator) sorted order to the thrift cas() method. -- Sylvain On Sun, Jul 21, 2013 at 1:22 PM, Soumava Ghosh wrote: > Hi, > > I'm taking a look at the Check and Set functionalities provided by the > cas() API provided by cassandra 2.0 (the code available on git). I'm > running a few tests on a small sized cluster (replication factor 3, > consistency level quorum) with a few clients. I've observed a lot of cases > seem to hit the TimedOutException while paxos is in progress. On inspection > of the server side logs the following stack was seen: > > ERROR [Thread-582] 2013-07-21 01:17:46,169 CassandraDaemon.java (line 196) > Exception in thread Thread[Thread-582,5,main] > java.lang.AssertionError: Added column does not sort as the last column > at > org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115) > at > org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:119) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:91) > at > org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:139) > at > org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:128) > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:175) > at > org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:135) > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) > > I was updating multiple columns using the same cas() call, and looking at > the assert (code below) I added code to sort them before sending. > However, now I'm seeing the same issue *even with single column* cas() > calls. > > int c = internalComparator().compare(columns.get(getColumnCount() > - 1).name(), column.name()); > // note that we want an assertion here (see addColumn javadoc), > but we also want that if > // assertion are disabled, addColumn works correctly with unsorted > input > assert c <= 0 : "Added column does not sort as the " + (reversed ? > "first" : "last") + " column"; > > I have also seen this error occur in the getRow() path, as below: > > java.lang.AssertionError: Added column does not sort as the last column > at > org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115) > at > org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117) > at > org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151) > at > org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:95) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:57) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1452) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1281) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195) > at org.apache.cassandra.db.Table.getRow(Table.java:331) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1288) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1813) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > > Could someone provide me a little more clarity about what the context of > the error is and what the client needs to do to fix it? I understand that > running with assertions disabled would do away with this error, but i'd > like to know if there is a proper way to fix this. Moreover, what could > possibly be going wrong in the get path that is hitting this issue? > > Some guidance, on the fact that whether it is expected that the client > sort the columns before calling the API with multiple columns, would be > really appreciated. > > Thanks, > Soumava > --e89a8ff1ca5263558904e2146463 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
This is a bug really:=A0https://issues.apache.org/jira/browse/CASSAN= DRA-5786.

This should get fixed in the next beta of = 2.0, but if you really want to test CAS updates in the meantime, you'll= have to provide the columns in (column family comparator) sorted order to = the thrift cas() method.

--
Sylvain


On Sun, Jul 21, 2013 at 1:22 PM, Sou= mava Ghosh <soumava@cs.utexas.edu> wrote:
Hi,

I= 9;m taking a look at the Check and Set functionalities provided by the cas(= ) API provided by cassandra 2.0 (the code available on git). I'm runnin= g a few tests on a small sized cluster (replication factor 3, consistency l= evel quorum) with a few clients. I've observed a lot of cases seem to h= it the TimedOutException while paxos is in progress. On inspection of the s= erver side logs the following stack was seen:

ERROR [Thread-582] 2013-07-21 01:17:46,169 Cassand= raDaemon.java (line 196) Exception in thread Thread[Thread-582,5,main]
java.lang.AssertionError: Added column does not sort as the last colu= mn
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.ArrayBackedSortedColumns.ad= dColumn(ArrayBackedSortedColumns.java:115)
=A0 =A0 =A0 =A0 at org= .apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.ColumnFamilySerializer.deseria= lize(ColumnFamilySerializer.java:119)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.ColumnFamilySerializer.dese= rialize(ColumnFamilySerializer.java:96)
=A0 =A0 =A0 =A0 at org.ap= ache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer= .java:91)
=A0 =A0 =A0 =A0 at org.apache.cassandra.service.paxos.Commit$CommitSer= ializer.deserialize(Commit.java:139)
=A0 =A0 =A0 =A0 at org.apach= e.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:1= 28)
=A0 =A0 =A0 =A0 at org.apache.cassandra.net.MessageIn.read(MessageIn.java:9= 9)
=A0 =A0 =A0 =A0 at org.apache.cassandra.net.IncomingTcpConnect= ion.receiveMessage(IncomingTcpConnection.java:175)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(I= ncomingTcpConnection.java:135)
=A0 =A0 =A0 =A0 at org.apache.cassandra.net.IncomingTcpConnection.run(= IncomingTcpConnection.java:82)

I was updating mult= iple columns using the same cas() call, and looking at the assert (code bel= ow) I added code to sort them before sending.=A0
However, now I'm seeing the same issue even=A0with=A0single col= umn cas() calls.=A0

=A0 =A0 =A0 =A0 int c =3D internalComparator().compare(col= umns.get(getColumnCount() - 1).name(), column.name());
=A0 =A0 =A0 = =A0 // note that we want an assertion here (see addColumn javadoc), but we = also want that if
=A0 =A0 =A0 =A0 // assertion are disabled, addColumn works = correctly with unsorted input
=A0 =A0 =A0 =A0 assert c <=3D= 0 : "Added column does not sort as the " + (reversed ? "fir= st" : "last") + " column";

I have also s= een this error occur in the getRow() path, as below:

java.lang.AssertionError: Added column does not so= rt as the last column
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBack= edSortedColumns.java:115)
=A0 =A0 = =A0 =A0 at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java= :117)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151)<= /span>
=A0 =A0 =A0 =A0 at org.apache.cass= andra.db.CollationController.collectTimeOrderedData(CollationController.jav= a:95)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.CollationController.getTopLevelColumns(Colla= tionController.java:57)<= br>=A0 =A0 = =A0 =A0 at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(Col= umnFamilyStore.java:1452)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFami= lyStore.java:1281)
=A0 =A0 =A0 =A0= at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyS= tore.java:1195)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.Table.getRow(Table.java:331)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.SliceB= yNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayTh= row(StorageProxy.java:1288)
=A0 = =A0 =A0 =A0 at org.apache.cassandra.service.StorageProxy$DroppableRunnable.= run(StorageProxy.java:1813)
=A0 =A0 =A0 = =A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1145)
=A0 =A0 =A0 =A0 at jav= a.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615= )
=A0 =A0 =A0 = =A0 at java.lang.Thread.run(Thread.java:722)

Could someone prov= ide me a little more clarity about what the context of the error is and wha= t the client needs to do to fix it? I understand that running with assertio= ns disabled would do away with this error, but i'd like to know if ther= e is a proper way to fix this. Moreover, what could possibly be going wrong= in the get path that is hitting this issue?

Some guidance, on the fact that whether it is exp= ected that the client sort the columns before calling the API with multiple= columns, would be really appreciated.

Thanks,
Soumava

--e89a8ff1ca5263558904e2146463--