incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thor Carpenter <>
Subject Re: 0.7RC1 local_quorum -> TimedOutException
Date Wed, 29 Dec 2010 20:38:44 GMT
In an attempt to replicate this error on a simpler system I have come across a different error
but seemingly related problem.  Now I am getting the following exception in system.log.

DEBUG [pool-1-thread-3] 2010-12-29 12:10:38,897 (line 362) insert
ERROR [pool-1-thread-3] 2010-12-29 12:10:38,906 (line 2960) Internal error
processing insert
        at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(
        at org.apache.cassandra.locator.TokenMetadata.ringIterator(
        at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(
        at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(
        at org.apache.cassandra.service.StorageService.getNaturalEndpoints(
        at org.apache.cassandra.service.StorageService.getNaturalEndpoints(
        at org.apache.cassandra.service.StorageProxy.mutate(
        at org.apache.cassandra.thrift.CassandraServer.doInsert(
        at org.apache.cassandra.thrift.CassandraServer.insert(
        at org.apache.cassandra.thrift.Cassandra$Processor$insert.process(
        at org.apache.cassandra.thrift.Cassandra$Processor.process(
        at org.apache.cassandra.thrift.CustomTThreadPoolServer$
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
        at java.util.concurrent.ThreadPoolExecutor$

Looking at, the assert that is triggered is:
assert ring.size() > 0;

Which led me to run nodetool ring which outputs a seemingly correct config:
-sh-3.2$ nodetool -h localhost -p 8080 ring
Address         Status State   Load            Owns    Token      Up     Normal  47.16 KB        100.00% 1

To reproduce:

* I have upgraded to 0.7RC3 so install the latest riptano 0.7rc3 rpm on a single node in "DC1".

* In cassandra.yaml set initial_token = 1


* Schema loaded via cassandra-cli:

create keyspace KeyspaceDC1 with
    replication_factor = 1 and
    placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and
    strategy_options = [{DC1:1, DC2:0}];

use KeyspaceDC1;

create column family TestCF with
    column_type = 'Standard' and
    comparator = 'BytesType' and
    keys_cached = 200000 and
    rows_cached = 2000 and
    gc_grace = 0 and
    read_repair_chance = 0.0;

* In cassandra-cli execute the following:

[default@unknown] use KeyspaceDC1;
Authenticated to keyspace: KeyspaceDC1
[default@KeyspaceDC1] set TestCF['some key']['some col'] = 'some value';
Internal error processing insert

* Check system.log where you should find the above assertion error.

Can anybody find a flaw in this setup?  Do others agree something looks buggy?


On Dec 28, 2010, at 4:33 PM, Thor Carpenter wrote:

Third try for good luck.  Maybe this time I'll get it all in one email.  :-)

At time 15:55 I performed a write with CL = ONE, a subsequent read with CL = ONE and received
correct results.  Similar results occur with any read CL other than LOCAL_QUORUM.

At time 15:58 I performed a write with CL = ONE, a subsequent read with CL = LOCAL_QUORUM
and received a TimedOutException.

System.log and cassandra.log are from server dlc05-sea to corresponds with the hector error
message "Closing client CassandraClient<dlc05-sea:9160-50>".  I haven't found anything
in the logs that indicates the problem but maybe others with more discerning eyes can.

Please let me know if I can provide any additional information.



View raw message