cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bialecki <andrew.biale...@klaviyo.com>
Subject Re: cassandra python driver routing requests to one node?
Date Tue, 15 Nov 2016 07:57:13 GMT
Is the node selection based on key deterministic across multiple clients?
If it is, that sounds plausible. For this particular workload it's
definitely possible to have a hot key / spot, but it was surprising it
wasn't three nodes that got hot, it was just one.

On Mon, Nov 14, 2016 at 6:26 PM, Alex Popescu <alexp@datastax.com> wrote:

> I'm wondering if what you are seeing is https://datastax-oss.
> atlassian.net/browse/PYTHON-643 (that could still be a sign of a
> potential data hotspot)
>
> On Sun, Nov 13, 2016 at 10:57 PM, Andrew Bialecki <
> andrew.bialecki@klaviyo.com> wrote:
>
>> We're using the "default" TokenAwarePolicy. Our nodes are spread across
>> different racks within one datacenter. I've turned on debug logging for the
>> Python driver, but it doesn't look like it logs which Casandra node each
>> request goes to, but maybe I haven't got the right logging set to debug.
>>
>> On Mon, Nov 14, 2016 at 12:39 AM, Ben Slater <ben.slater@instaclustr.com>
>> wrote:
>>
>>> What load balancing policies are you using in your client code (
>>> https://datastax.github.io/python-driver/api/cassandra/policies.html)?
>>>
>>> Cheers
>>> Ben
>>>
>>> On Mon, 14 Nov 2016 at 16:22 Andrew Bialecki <
>>> andrew.bialecki@klaviyo.com> wrote:
>>>
>>>> We have an odd situation where all of a sudden of our cluster started
>>>> seeing a disproportionate number of writes go to one node. We're using the
>>>> Python driver version 3.7.1. I'm not sure if this is a driver issue or
>>>> possibly a network issue causing requests to get routed in an odd way. It's
>>>> not absolute, there are requests going to all nodes.
>>>>
>>>> Tried restarting the problematic node, no luck (those are the quiet
>>>> periods). Tried restarting the clients, also no luck. Checked nodetool
>>>> status and ownership is even across the cluster.
>>>>
>>>> Curious if anyone's seen this behavior before. Seems like the next step
>>>> will be to debug the client and see why it's choosing that node.
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>> --
>>>> AB
>>>>
>>>
>>
>>
>> --
>> AB
>>
>
>
>
> --
> Bests,
>
> Alex Popescu | @al3xandru
> Sen. Product Manager @ DataStax
>
>
>
>


-- 
AB

Mime
View raw message