Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C3CC2104FD for ; Mon, 22 Jul 2013 08:43:26 +0000 (UTC) Received: (qmail 28885 invoked by uid 500); 22 Jul 2013 08:43:24 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 28861 invoked by uid 500); 22 Jul 2013 08:43:23 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 28850 invoked by uid 99); 22 Jul 2013 08:43:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jul 2013 08:43:23 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of soumava@utexas.edu designates 209.85.216.49 as permitted sender) Received: from [209.85.216.49] (HELO mail-qa0-f49.google.com) (209.85.216.49) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jul 2013 08:43:19 +0000 Received: by mail-qa0-f49.google.com with SMTP id cr7so836391qab.1 for ; Mon, 22 Jul 2013 01:42:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:content-type :x-gm-message-state; bh=sf1he6RIKwlKdyfpQhChC5AwGisY4UBzP7/KAIv7KGI=; b=LZvdYF0l+kbTjYI+TwifCgbYkTDzm+HXA/o/qvPJCznd0kyNJ1s1ecp+psV+9xL4xO muGDrRvszWnEZhTjeF604hB6h9XxlV2mddLQXicaIsmUXUjgiI+JodhLk2ZBMND1m0vB NX+pWnS1Ul11YdyPMy2KyLY6svHfLQ3yPP5J5zEP4VXMqp9y1eGmUF/UvasImukNIz1R 4uFqNnXPQfxKoWZvTIJARPjzdRVyvmMUu/+Y87V/4CNh2zGrEyK/ZRWuD39sbGIVirIX mzEixgOMGsOOwPvP25l9+X+XgAdWtt3Ryhd0mUadZgNzKCmYu9QCpWv+VmIuRqS+XN4g oqsw== X-Received: by 10.49.34.206 with SMTP id b14mr12206782qej.54.1374482578549; Mon, 22 Jul 2013 01:42:58 -0700 (PDT) MIME-Version: 1.0 Sender: soumava@utexas.edu Received: by 10.49.105.168 with HTTP; Mon, 22 Jul 2013 01:42:38 -0700 (PDT) In-Reply-To: References: From: Soumava Ghosh Date: Mon, 22 Jul 2013 01:42:38 -0700 X-Google-Sender-Auth: qHnn2Ti2qkrXZgv30K0Ix0ypYUk Message-ID: Subject: Re: Cassandra 2.0 - AssertionError in ArrayBackedSortedColumns To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b414222d8411e04e215a96e X-Gm-Message-State: ALoCoQk5voZKy+QTQw1g4WSBJ1yMhYb14WU8PiF+l9QtG4WyYZvx1rJB4ITohQ1dm8b8sfzA7Eh4 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b414222d8411e04e215a96e Content-Type: text/plain; charset=ISO-8859-1 Thanks for the confirmation! It looks like the code is capable of handling the data in unsorted order with assertions disabled; so for the time being I will disable the assertion for my tests, until the fixes are available. Thanks, Soumava On Mon, Jul 22, 2013 at 1:33 AM, Sylvain Lebresne wrote: > Actually, your second stack trace is due to > https://issues.apache.org/jira/browse/CASSANDRA-5788. > > > On Mon, Jul 22, 2013 at 9:37 AM, Sylvain Lebresne wrote: > >> >> unless data was stored in incorrect order and I don't know if that is >>> possible through the API.. >>> >> >> That's exactly what happens and that's why I say it's a bug. >> >> >>> >>> ii. I am also seeing some sporadic cases where a single column cas() >>> update is hitting this same issue (the bug you mentioned) as multiple >>> columns, do you see any reason for that happening? >>> >> >> That would be slightly weirder, but frankly that's almost surely due to >> https://issues.apache.org/jira/browse/CASSANDRA-5786 too. >> >> -- >> Sylvain >> >> >>> Thank you, >>> Soumava >>> >>> >>> On Mon, Jul 22, 2013 at 12:11 AM, Sylvain Lebresne >> > wrote: >>> >>>> This is a bug really: >>>> https://issues.apache.org/jira/browse/CASSANDRA-5786. >>>> >>>> This should get fixed in the next beta of 2.0, but if you really want >>>> to test CAS updates in the meantime, you'll have to provide the columns in >>>> (column family comparator) sorted order to the thrift cas() method. >>>> >>>> -- >>>> Sylvain >>>> >>>> >>>> On Sun, Jul 21, 2013 at 1:22 PM, Soumava Ghosh wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm taking a look at the Check and Set functionalities provided by the >>>>> cas() API provided by cassandra 2.0 (the code available on git). I'm >>>>> running a few tests on a small sized cluster (replication factor 3, >>>>> consistency level quorum) with a few clients. I've observed a lot of cases >>>>> seem to hit the TimedOutException while paxos is in progress. On inspection >>>>> of the server side logs the following stack was seen: >>>>> >>>>> ERROR [Thread-582] 2013-07-21 01:17:46,169 CassandraDaemon.java (line >>>>> 196) Exception in thread Thread[Thread-582,5,main] >>>>> java.lang.AssertionError: Added column does not sort as the last column >>>>> at >>>>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115) >>>>> at >>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117) >>>>> at >>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:119) >>>>> at >>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:96) >>>>> at >>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:91) >>>>> at >>>>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:139) >>>>> at >>>>> org.apache.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:128) >>>>> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) >>>>> at >>>>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:175) >>>>> at >>>>> org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:135) >>>>> at >>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) >>>>> >>>>> I was updating multiple columns using the same cas() call, and looking >>>>> at the assert (code below) I added code to sort them before sending. >>>>> However, now I'm seeing the same issue *even with single column*cas() calls. >>>>> >>>>> int c = >>>>> internalComparator().compare(columns.get(getColumnCount() - 1).name(), >>>>> column.name()); >>>>> // note that we want an assertion here (see addColumn >>>>> javadoc), but we also want that if >>>>> // assertion are disabled, addColumn works correctly with >>>>> unsorted input >>>>> assert c <= 0 : "Added column does not sort as the " + >>>>> (reversed ? "first" : "last") + " column"; >>>>> >>>>> I have also seen this error occur in the getRow() path, as below: >>>>> >>>>> java.lang.AssertionError: Added column does not sort as the last column >>>>> at >>>>> org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115) >>>>> at >>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117) >>>>> at >>>>> org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151) >>>>> at >>>>> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:95) >>>>> at >>>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:57) >>>>> at >>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1452) >>>>> at >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1281) >>>>> at >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195) >>>>> at org.apache.cassandra.db.Table.getRow(Table.java:331) >>>>> at >>>>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55) >>>>> at >>>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1288) >>>>> at >>>>> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1813) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:722) >>>>> >>>>> Could someone provide me a little more clarity about what the context >>>>> of the error is and what the client needs to do to fix it? I understand >>>>> that running with assertions disabled would do away with this error, but >>>>> i'd like to know if there is a proper way to fix this. Moreover, what could >>>>> possibly be going wrong in the get path that is hitting this issue? >>>>> >>>>> Some guidance, on the fact that whether it is expected that the client >>>>> sort the columns before calling the API with multiple columns, would be >>>>> really appreciated. >>>>> >>>>> Thanks, >>>>> Soumava >>>>> >>>> >>>> >>> >> > --047d7b414222d8411e04e215a96e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks for the confirmation!

It looks like th= e code is capable of handling the data in unsorted order with assertions di= sabled; so for the time being I will disable the assertion for my tests, un= til the fixes are available.=A0

Thanks,
Soumava


On Mon, Jul 22, 201= 3 at 1:33 AM, Sylvain Lebresne <sylvain@datastax.com> wro= te:
Actually, your second stack= trace is due to=A0https://issues.apache.org/jira/browse/CASSANDRA-= 5788.


=
On Mon, Jul 22, 2013 at 9:37 AM, Sylvain Lebresne <sylvain@datastax.com= > wrote:

=
=A0unless data was stored in incorrect order and I do= n't know if that is possible through the API..
=

That's exactly what happens and that's wh= y I say it's a bug.
=A0

ii. I am also seeing some sporadic cases where a single= column cas() update is hitting this same issue (the bug you mentioned) as = multiple columns, do you see any reason for that happening?

That would be slightly weirder, but = frankly that's almost surely due to https://issues.apache.org/j= ira/browse/CASSANDRA-5786 too.

--
Sylvain


Thank you,
Soumava


On Mon, Jul 22, 20= 13 at 12:11 AM, Sylvain Lebresne <sylvain@datastax.com> w= rote:
This is a bug really:=A0https://i= ssues.apache.org/jira/browse/CASSANDRA-5786.

This should get fixed in the next beta of 2.0, but if you re= ally want to test CAS updates in the meantime, you'll have to provide t= he columns in (column family comparator) sorted order to the thrift cas() m= ethod.

--
Sylvain


On Sun, Jul 21, 2013 at 1:= 22 PM, Soumava Ghosh <soumava@cs.utexas.edu> wrote:
Hi,

I'm taking a l= ook at the Check and Set functionalities provided by the cas() API provided= by cassandra 2.0 (the code available on git). I'm running a few tests = on a small sized cluster (replication factor 3, consistency level quorum) w= ith a few clients. I've observed a lot of cases seem to hit the TimedOu= tException while paxos is in progress. On inspection of the server side log= s the following stack was seen:

ERROR [Thread-582] 2013-07-21 01:17:46,169 Cassand= raDaemon.java (line 196) Exception in thread Thread[Thread-582,5,main]
java.lang.AssertionError: Added column does not sort as the last colu= mn
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.ArrayBackedSortedColumns.ad= dColumn(ArrayBackedSortedColumns.java:115)
=A0 =A0 =A0 =A0 at org= .apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:117)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.ColumnFamilySerializer.deseria= lize(ColumnFamilySerializer.java:119)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.ColumnFamilySerializer.dese= rialize(ColumnFamilySerializer.java:96)
=A0 =A0 =A0 =A0 at org.ap= ache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer= .java:91)
=A0 =A0 =A0 =A0 at org.apache.cassandra.service.paxos.Commit$CommitSer= ializer.deserialize(Commit.java:139)
=A0 =A0 =A0 =A0 at org.apach= e.cassandra.service.paxos.Commit$CommitSerializer.deserialize(Commit.java:1= 28)
=A0 =A0 =A0 =A0 at org.apache.cassandra.net.MessageIn.read(MessageIn.java:9= 9)
=A0 =A0 =A0 =A0 at org.apache.cassandra.net.IncomingTcpConnect= ion.receiveMessage(IncomingTcpConnection.java:175)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(I= ncomingTcpConnection.java:135)
=A0 =A0 =A0 =A0 at org.apache.cassandra.net.IncomingTcpConnection.run(= IncomingTcpConnection.java:82)

I was updating mult= iple columns using the same cas() call, and looking at the assert (code bel= ow) I added code to sort them before sending.=A0
However, now I'm seeing the same issue even=A0with=A0single col= umn cas() calls.=A0

=A0 =A0 =A0 =A0 int c =3D internalComparator().compare(col= umns.get(getColumnCount() - 1).name(), column.name());
=A0 =A0 =A0 = =A0 // note that we want an assertion here (see addColumn javadoc), but we = also want that if
=A0 =A0 =A0 =A0 // assertion are disabled, addColumn works = correctly with unsorted input
=A0 =A0 =A0 =A0 assert c <=3D= 0 : "Added column does not sort as the " + (reversed ? "fir= st" : "last") + " column";

I have also s= een this error occur in the getRow() path, as below:

java.lang.AssertionError: Added column does not so= rt as the last column
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBack= edSortedColumns.java:115)
=A0 =A0 = =A0 =A0 at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java= :117)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:151)<= /span>
=A0 =A0 =A0 =A0 at org.apache.cass= andra.db.CollationController.collectTimeOrderedData(CollationController.jav= a:95)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.CollationController.getTopLevelColumns(Colla= tionController.java:57)<= br>=A0 =A0 = =A0 =A0 at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(Col= umnFamilyStore.java:1452)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFami= lyStore.java:1281)
=A0 =A0 =A0 =A0= at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyS= tore.java:1195)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.db.Table.getRow(Table.java:331)
=A0 =A0 =A0 =A0 at org.apache.cassandra.db.SliceB= yNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
=A0 =A0 =A0 = =A0 at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayTh= row(StorageProxy.java:1288)
=A0 = =A0 =A0 =A0 at org.apache.cassandra.service.StorageProxy$DroppableRunnable.= run(StorageProxy.java:1813)
=A0 =A0 =A0 = =A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1145)
=A0 =A0 =A0 =A0 at jav= a.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615= )
=A0 =A0 =A0 = =A0 at java.lang.Thread.run(Thread.java:722)

Could someone prov= ide me a little more clarity about what the context of the error is and wha= t the client needs to do to fix it? I understand that running with assertio= ns disabled would do away with this error, but i'd like to know if ther= e is a proper way to fix this. Moreover, what could possibly be going wrong= in the get path that is hitting this issue?

Some guidance, on the fact that whether it is exp= ected that the client sort the columns before calling the API with multiple= columns, would be really appreciated.

Thanks,
Soumava





--047d7b414222d8411e04e215a96e--