cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Serediuk, Adam" <Adam.Sered...@serialssolutions.com>
Subject Re: Range Slice Issue
Date Tue, 03 May 2011 18:02:31 GMT
Both data and system load are equal across all nodes and the smaller test cluster also exhibits
the same issue. tokens are balanced and total node size is equivalent.

On May 3, 2011, at 10:51 AM, Jonathan Ellis wrote:

> So either (a) dynamic snitch is wrong or (b) those nodes really are
> more heavily loaded than the others, and are correctly pushing queries
> to other replicas.
> 
> On Tue, May 3, 2011 at 12:47 PM, Serediuk, Adam
> <Adam.Serediuk@serialssolutions.com> wrote:
>> I just ran a test and we do not see that behavior with dynamic snitch disabled. All
nodes appear to be doing local reads as expected.
>> 
>> 
>> On May 3, 2011, at 10:37 AM, Jonathan Ellis wrote:
>> 
>>> Do you still see this behavior if you disable dynamic snitch?
>>> 
>>> On Tue, May 3, 2011 at 12:31 PM, Serediuk, Adam
>>> <Adam.Serediuk@serialssolutions.com> wrote:
>>>> We appear to have encountered an issue with cassandra 0.7.5 after upgrading
>>>> from 0.7.2. While doing a batch read using a get_range_slice against the
>>>> ranges an individual node is master for we are able to reproduce
>>>> consistently that the last two nodes in the ring, regardless of the ring
>>>> size (we have a 60 node production cluster and a 12 node test cluster)
>>>> perform this read over the network using replicas of executing locally.
>>>> Every other node in the ring successfully reads locally.
>>>> To be sure there were no data consistency issues we performed a nodetool
>>>> repair against both of these nodes and the issue persists. We also tried
>>>> truncating the column family and repopulating, but the issue remains.
>>>> This seems to be related to CASSANDRA-2286 in 0.7.4. We always want to read
>>>> data locally if it is available there. We
>>>> use Cassandra.Client.describe_ring() to figure out which machine in the
>>>> ring is master for which TokenRange. I then compare the master for
>>>> each TokenRange against the localhost to find out which token ranges
>>>> are owned by the local machine (remote reads are too slow for this type
>>>> of batch processing). Once I know which TokenRanges are on
>>>> each machine locally I get evenly sized splits using
>>>> Cassandra.Client.describe_splits().
>>>> 
>>>> Adam
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of DataStax, the source for professional Cassandra support
>>> http://www.datastax.com
>>> 
>> 
>> 
>> 
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
> 



Mime
View raw message