cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
Date Wed, 04 Jun 2014 22:09:02 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14018257#comment-14018257
] 

Benedict edited comment on CASSANDRA-7245 at 6/4/14 10:08 PM:
--------------------------------------------------------------

[~jasobrown] I got this one if you've got better stuff to do :-)

[~tjake]: 
* I think the change you made doesn't actually totally eliminate the bug: since it extends
DroppableRunnable, the finally block may never be run
* I think it would be good in SP.mutate() you can't release the mutations until after we get
the all clear from the replicas, as we may have to use the mutations to write local hints.
It might be nice, however, to have the response handler release() the reference once we receive
enough responses from our replicas, so that we don't keep all of the references if we're just
waiting for one mutation
* Why do we need a ThreadLocal in ClientState to store the sourceFrame. Can't we just store
it directly in a field in QueryState?

Nit:
* WS in ClientState constructor; would also prefer to just create ThreadLocal in var initialiser
since it's always init'd empty

Also, this is off-topic, but I wonder if we shouldn't replace the NBHM in ServerConnection
with a simple array. We know the range is < 32K (<.1K for older clients), and each index
is accessed by a single thread at any given time, so we'd just need to be able to atomically
swap in larger arrays if we wanted dynamic sizing. 

Otherwise LGTM


was (Author: benedict):
[~jasobrown] I got this one if you've got better stuff to do :-)

[~tjke]: 
* I think the change you made doesn't actually totally eliminate the bug: since it extends
DroppableRunnable, the finally block may never be run
* I think it would be good in SP.mutate() you can't release the mutations until after we get
the all clear from the replicas, as we may have to use the mutations to write local hints.
It might be nice, however, to have the response handler release() the reference once we receive
enough responses from our replicas, so that we don't keep all of the references if we're just
waiting for one mutation
* Why do we need a ThreadLocal in ClientState to store the sourceFrame. Can't we just store
it directly in a field in QueryState?

Nit:
* WS in ClientState constructor; would also prefer to just create ThreadLocal in var initialiser
since it's always init'd empty

Also, this is off-topic, but I wonder if we shouldn't replace the NBHM in ServerConnection
with a simple array. We know the range is < 32K (<.1K for older clients), and each index
is accessed by a single thread at any given time, so we'd just need to be able to atomically
swap in larger arrays if we wanted dynamic sizing. 

Otherwise LGTM

> Out-of-Order keys with stress + CQL3
> ------------------------------------
>
>                 Key: CASSANDRA-7245
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Pavel Yaskevich
>            Assignee: T Jake Luciani
>             Fix For: 2.1.0
>
>         Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, 7245v4.txt,
netty-all-4.0.19.Final.jar
>
>
> We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found
following problem almost in every SSTable generated (~200 GB of data and 821 SSTables).
> We set up they keys to be 10 bytes in size (default) and population between 1 and 600000000.
> Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions:
> _There is a problem with sorting of normal looking keys:_
> 30303039443538353645
> 30303039443745364242
> java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*)
> DecoratedKey(-1767746583617597213, *30303039443437454333*)
> 00000a30303033343933
> 37344400001388343933
> java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *37344400001388343933*)
> DecoratedKey(-7565486415339257200, *30303033344639443137*)
> 30303033354244363031
> 30303033354133423742
> java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*)
> DecoratedKey(-7838239767410066684, *30303033354145344534*)
> 30303034313442354137
> 30343136353633340000
> java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *30343136353633340000*)
> DecoratedKey(-9106177395653818217, *30303034313333444238*)
> 30303035373044373435
> 30303035373044334631
> java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*)
> DecoratedKey(-4296696226469000945, *30303035373132364138*)
> _And completely different ones:_
> 30303041333745373543
> 7cd045c59a90d7587d8d
> java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*)
> DecoratedKey(-5146766422778260690, *30303041333943303232*)
> 30303033323144444144
> 30303033323346343932
> java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, *30303033323346343932*)
> DecoratedKey(5233296131921119414, *53d83e00000012287e03*)
> 30303034314531374431
> 3806734b256c27e41ec2
> java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, *3806734b256c27e41ec2*)
> DecoratedKey(-8072288379146044663, *30303034314136413343*)
> _And sometimes there is no problem at all:_
> 30303033353144463637
> 0000002a31b3b31a1c2f
> 5d616dd38211ebb5d6ec
> 44423645000013880000
> 00001388138844463744
> 30303033353143394343
> It's worth to mention that we have got 22 timeout exceptions but number of out-of-order
keys is much larger than that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message