incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roland Gude <roland.g...@yoochoose.com>
Subject AW: IndexSliceQuery issue - ReadStage piling up (looks like deadlock/infinite loop or similar)
Date Thu, 11 Aug 2011 10:15:18 GMT
Yes, i can reproduce this behavior

If issue a query like this (on 0.7.8 with patch for CASSANDRA-2964 applied)
[default@demo]get users where birth_date = 1968 and state = 'UT';
with an index on birth_date but no index on state
I do not get results (actually I get '0 rows') even though there are rows which statisfy all
clauses.
However, if I repeat this several times several of the nodes start piling up pending reads.
(tpstats shows some 80000 reads pending)
And even though the nodes are not able to fulfill (read) requests anymore, they are not marked
as down by the gossiper. Overall this results in an unusable cluster.

If I do the same thing on a 0.7.5 cluster
Cassandra logs a nullpointerexception and the cli returns with null, but the cluster stays
functional.


Von: aaron morton [mailto:aaron@thelastpickle.com]
Gesendet: Mittwoch, 10. August 2011 23:48
An: user@cassandra.apache.org
Betreff: Re: IndexSliceQuery issue - ReadStage piling up (looks like deadlock/infinite loop
or similar)

Are you still having a problem ? I'm a bit confused about what you saying.

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 10 Aug 2011, at 03:33, Roland Gude wrote:


Hi,

I experience issues when doing a indexslicequery with multiple expressions if one of the expressions
is about a non index column

I did the equivalent of this example (but with my data) from
http://www.datastax.com/dev/blog/whats-new-cassandra-07-secondary-indexes


Secondary indexes automate this. Let's add some state data:

[default@demo] set users[bsanderson][state] = 'UT';
[default@demo] set users[prothfuss][state] = 'WI';
[default@demo] set users[htayler][state] = 'UT';

Note that even though state is not indexed yet, we can include the new state data in a query
as long as another column in the query is indexed:

[default@demo] get users where state = 'UT';
No indexed columns present in index clause with operator EQ
[default@demo] get users where state = 'UT' and birth_date > 1970;
No indexed columns present in index clause with operator EQ
[default@demo]get users where birth_date = 1968 and state = 'UT';
-------------------
RowKey: htayler
=> (column=birth_date, value=1968, timestamp=1291334765649000)
=> (column=full_name, value=Howard Tayler, timestamp=1291334749160000)
=> (column=state, value=5554, timestamp=1291334890708000)

On On 0.7.8 (with CASSANDRA-2964 applied)
This example will not return any data, but return "0 rows". I repeated the query multiple
times with different variations for the values which should all have returned data, but eventually
I ended up with the cluster having 80000 reads pending on some of the nodes

On 0.7.5 the query will result in a NullPointerException being thrown and "null" returned
in the cli

ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java (line 113) Fatal
exception in thread Thread[ReadStage:258,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
        at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
        at org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
        ... 4 more
ERROR [ReadStage:258] 2011-08-09 16:03:27,153 AbstractCassandraDaemon.java (line 113) Fatal
exception in thread Thread[ReadStage:258,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:51)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
        at org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
        at org.apache.cassandra.db.ColumnFamilyStore.scan(ColumnFamilyStore.java:1615)
        at org.apache.cassandra.service.IndexScanVerbHandler.doVerb(IndexScanVerbHandler.java:42)
        ... 4 more



Can anybody reproduce this?

Greetings,
Roland

--
YOOCHOOSE GmbH

Roland Gude
Software Engineer

Im Mediapark 8, 50670 Köln

+49 221 4544151 (Tel)
+49 221 4544159 (Fax)
+49 171 7894057 (Mobil)


Email: roland.gude@yoochoose.com<mailto:roland.gude@yoochoose.com>
WWW: www.yoochoose.com<http://www.yoochoose.com/>

YOOCHOOSE GmbH
Geschäftsführer: Dr. Uwe Alkemper, Michael Friedmann
Handelsregister: Amtsgericht Köln HRB 65275
Ust-Ident-Nr: DE 264 773 520
Sitz der Gesellschaft: Köln



Mime
View raw message