cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anishek Agarwal <anis...@gmail.com>
Subject Re: Traffic inconsistent across nodes
Date Mon, 18 Apr 2016 09:16:02 GMT
OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent
OS 7)
All of them are running Cassandra 2.0.17

output of the test :

host ip: 10.124.114.113

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.108

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.110

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.118

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.125.138.59

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.97

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.105

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17

host ip: 10.124.114.98

host DC : WDC

distance of host: LOCAL

host is up: true

cassandra version : 2.0.17


On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens <mightye@gmail.com> wrote:

> Thanks for that, that helps a lot.  The next thing to check might be
> whether or not your application actually has access to the other nodes.
> With that topology, and assuming all the nodes you included in your
> original graph are in the 'WDC' data center, I'd be inclined to look for a
> network issue of some kind.
>
> Also, it probably doesn't matter, but what OS / Distribution are you
> running the servers and clients on?
>
> Check with netcat or something that you can reach all the configured ports
> from your application server, but also the driver itself offers some
> introspection into its view of individual connection health.  This is a
> little bit ugly, but this is how we include information about connection
> status in an API for health monitoring from a Scala application using the
> Java driver; hopefully you can use it to see how to access information
> about the driver's view of host health from the application's perspective.
> Most importantly I'd suggest looking for host.isUp status and
> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
> in your target datacenter to be LOCAL.
>
> "hosts" -> {
>   val hosts: Map[String, Map[String, mutable.Set[Host]]] =
>     connection.getMetadata
>       .getAllHosts.asScala
>       .groupBy(_.getDatacenter)
>       .mapValues(_.groupBy(_.getRack))
>   val lbp: LoadBalancingPolicy = connection.getConfiguration.getPolicies.getLoadBalancingPolicy
>   JsObject(hosts.map { case (dc: String, rackAndHosts) =>
>     dc -> JsObject(rackAndHosts.map { case (rack: String, hosts: mutable.Set[Host])
=>
>       rack -> JsArray(hosts.map { host =>
>         Json.obj(
>           "address"          -> host.getAddress.toString,
>           "socketAddress"    -> host.getSocketAddress.toString,
>           "cassandraVersion" -> host.getCassandraVersion.toString,
>           "isUp"             -> host.isUp,
>           "hostDistance"     -> lbp.distance(host).toString
>         )
>       }.toSeq)
>     }.toSeq)
>   }.toSeq)
> },
>
>
> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal <anishek@gmail.com>
> wrote:
>
>> here is the output:  every node in a single DC is in the same rack.
>>
>> Datacenter: WDC5
>>
>> ================
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  Address         Load       Tokens  Owns (effective)  Host ID
>>                       Rack
>>
>> UN  10.125.138.33   299.22 GB  256     64.2%
>> 8aaa6015-d444-4551-a3c5-3257536df476  RAC1
>>
>> UN  10.125.138.125  329.38 GB  256     70.3%
>> 70be44a2-de17-41f1-9d3a-6a0be600eedf  RAC1
>>
>> UN  10.125.138.129  305.11 GB  256     65.5%
>> 0fbc7f44-7062-4996-9eba-2a05ae1a7032  RAC1
>>
>> Datacenter: WDC
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> --  Address         Load       Tokens  Owns (effective)  Host ID
>>                       Rack
>>
>> UN  10.124.114.105  151.09 GB  256     38.0%
>> c432357d-bf81-4eef-98e1-664c178a3c23  RAC1
>>
>> UN  10.124.114.110  150.15 GB  256     36.9%
>> 6f92d32e-1c64-4145-83d7-265c331ea408  RAC1
>>
>> UN  10.124.114.108  170.1 GB   256     41.3%
>> 040ae7e5-3f1e-4874-8738-45edbf576e12  RAC1
>>
>> UN  10.124.114.98   165.34 GB  256     37.6%
>> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c  RAC1
>>
>> UN  10.124.114.113  145.22 GB  256     35.7%
>> 1557af04-e658-4751-b984-8e0cdc41376e  RAC1
>>
>> UN  10.125.138.59   162.65 GB  256     38.6%
>> 9ba1b7b6-5655-456e-b1a1-6f429750fc96  RAC1
>>
>> UN  10.124.114.97   164.03 GB  256     36.9%
>> c918e497-498e-44c3-ab01-ab5cb4d48b09  RAC1
>>
>> UN  10.124.114.118  139.62 GB  256     35.1%
>> 2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9  RAC1
>>
>> On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens <mightye@gmail.com> wrote:
>>
>>> The output of nodetool status would really help answer some questions.
>>> I take it the 8 hosts in your graph are in the same DC.  Are the four
>>> serving writes in the same logical or physical rack (as Cassandra sees it),
>>> while the others are not?
>>>
>>> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <anishek@gmail.com>
>>> wrote:
>>>
>>>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>>>
>>>>
>>>>
>>>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <mightye@gmail.com>
>>>> wrote:
>>>>
>>>>> Maybe include nodetool status here?  Are the four nodes serving reads
>>>>> in one DC (local to your driver's config) while the others are in another?
>>>>>
>>>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <anishek@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> hello,
>>>>>>
>>>>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>>>>> across the nodes.
>>>>>>
>>>>>> its very surprising that only 4 nodes show transmitting (purple)
>>>>>> packets.
>>>>>>
>>>>>> our driver configuration on clients has the following load balancing
>>>>>> configuration  :
>>>>>>
>>>>>> new TokenAwarePolicy(
>>>>>>         new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME,
"WDC")),
>>>>>>         true)
>>>>>>
>>>>>>
>>>>>> any idea what is that we are missing which is leading to this skewed
>>>>>> data read patterns
>>>>>>
>>>>>> cassandra drivers as below:
>>>>>>
>>>>>> <dependency>
>>>>>>     <groupId>com.datastax.cassandra</groupId>
>>>>>>     <artifactId>cassandra-driver-core</artifactId>
>>>>>>     <version>2.1.6</version>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>>     <groupId>com.datastax.cassandra</groupId>
>>>>>>     <artifactId>cassandra-driver-mapping</artifactId>
>>>>>>     <version>2.1.6</version>
>>>>>> </dependency>
>>>>>>
>>>>>> cassandra version is 2.0.17
>>>>>>
>>>>>> Thanks in advance for the help.
>>>>>>
>>>>>> Anishek
>>>>>>
>>>>>>
>>>>
>>

Mime
View raw message