cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anishek Agarwal <anis...@gmail.com>
Subject Re: Traffic inconsistent across nodes
Date Tue, 19 Apr 2016 04:01:42 GMT
Looks like some problem with our monitoring framework. Thanks for you help !

On Mon, Apr 18, 2016 at 2:46 PM, Anishek Agarwal <anishek@gmail.com> wrote:

> OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent
> OS 7)
> All of them are running Cassandra 2.0.17
>
> output of the test :
>
> host ip: 10.124.114.113
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.108
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.110
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.118
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.125.138.59
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.97
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.105
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.98
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
>
> On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens <mightye@gmail.com> wrote:
>
>> Thanks for that, that helps a lot.  The next thing to check might be
>> whether or not your application actually has access to the other nodes.
>> With that topology, and assuming all the nodes you included in your
>> original graph are in the 'WDC' data center, I'd be inclined to look for a
>> network issue of some kind.
>>
>> Also, it probably doesn't matter, but what OS / Distribution are you
>> running the servers and clients on?
>>
>> Check with netcat or something that you can reach all the configured
>> ports from your application server, but also the driver itself offers some
>> introspection into its view of individual connection health.  This is a
>> little bit ugly, but this is how we include information about connection
>> status in an API for health monitoring from a Scala application using the
>> Java driver; hopefully you can use it to see how to access information
>> about the driver's view of host health from the application's perspective.
>> Most importantly I'd suggest looking for host.isUp status and
>> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
>> in your target datacenter to be LOCAL.
>>
>> "hosts" -> {
>>   val hosts: Map[String, Map[String, mutable.Set[Host]]] =
>>     connection.getMetadata
>>       .getAllHosts.asScala
>>       .groupBy(_.getDatacenter)
>>       .mapValues(_.groupBy(_.getRack))
>>   val lbp: LoadBalancingPolicy = connection.getConfiguration.getPolicies.getLoadBalancingPolicy
>>   JsObject(hosts.map { case (dc: String, rackAndHosts) =>
>>     dc -> JsObject(rackAndHosts.map { case (rack: String, hosts: mutable.Set[Host])
=>
>>       rack -> JsArray(hosts.map { host =>
>>         Json.obj(
>>           "address"          -> host.getAddress.toString,
>>           "socketAddress"    -> host.getSocketAddress.toString,
>>           "cassandraVersion" -> host.getCassandraVersion.toString,
>>           "isUp"             -> host.isUp,
>>           "hostDistance"     -> lbp.distance(host).toString
>>         )
>>       }.toSeq)
>>     }.toSeq)
>>   }.toSeq)
>> },
>>
>>
>> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal <anishek@gmail.com>
>> wrote:
>>
>>> here is the output:  every node in a single DC is in the same rack.
>>>
>>> Datacenter: WDC5
>>>
>>> ================
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> --  Address         Load       Tokens  Owns (effective)  Host ID
>>>                       Rack
>>>
>>> UN  10.125.138.33   299.22 GB  256     64.2%
>>> 8aaa6015-d444-4551-a3c5-3257536df476  RAC1
>>>
>>> UN  10.125.138.125  329.38 GB  256     70.3%
>>> 70be44a2-de17-41f1-9d3a-6a0be600eedf  RAC1
>>>
>>> UN  10.125.138.129  305.11 GB  256     65.5%
>>> 0fbc7f44-7062-4996-9eba-2a05ae1a7032  RAC1
>>>
>>> Datacenter: WDC
>>>
>>> ===============
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> --  Address         Load       Tokens  Owns (effective)  Host ID
>>>                       Rack
>>>
>>> UN  10.124.114.105  151.09 GB  256     38.0%
>>> c432357d-bf81-4eef-98e1-664c178a3c23  RAC1
>>>
>>> UN  10.124.114.110  150.15 GB  256     36.9%
>>> 6f92d32e-1c64-4145-83d7-265c331ea408  RAC1
>>>
>>> UN  10.124.114.108  170.1 GB   256     41.3%
>>> 040ae7e5-3f1e-4874-8738-45edbf576e12  RAC1
>>>
>>> UN  10.124.114.98   165.34 GB  256     37.6%
>>> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c  RAC1
>>>
>>> UN  10.124.114.113  145.22 GB  256     35.7%
>>> 1557af04-e658-4751-b984-8e0cdc41376e  RAC1
>>>
>>> UN  10.125.138.59   162.65 GB  256     38.6%
>>> 9ba1b7b6-5655-456e-b1a1-6f429750fc96  RAC1
>>>
>>> UN  10.124.114.97   164.03 GB  256     36.9%
>>> c918e497-498e-44c3-ab01-ab5cb4d48b09  RAC1
>>>
>>> UN  10.124.114.118  139.62 GB  256     35.1%
>>> 2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9  RAC1
>>>
>>> On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens <mightye@gmail.com> wrote:
>>>
>>>> The output of nodetool status would really help answer some questions.
>>>> I take it the 8 hosts in your graph are in the same DC.  Are the four
>>>> serving writes in the same logical or physical rack (as Cassandra sees it),
>>>> while the others are not?
>>>>
>>>> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <anishek@gmail.com>
>>>> wrote:
>>>>
>>>>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <mightye@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Maybe include nodetool status here?  Are the four nodes serving reads
>>>>>> in one DC (local to your driver's config) while the others are in
another?
>>>>>>
>>>>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <anishek@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> hello,
>>>>>>>
>>>>>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>>>>>> across the nodes.
>>>>>>>
>>>>>>> its very surprising that only 4 nodes show transmitting (purple)
>>>>>>> packets.
>>>>>>>
>>>>>>> our driver configuration on clients has the following load balancing
>>>>>>> configuration  :
>>>>>>>
>>>>>>> new TokenAwarePolicy(
>>>>>>>         new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME,
"WDC")),
>>>>>>>         true)
>>>>>>>
>>>>>>>
>>>>>>> any idea what is that we are missing which is leading to this
skewed
>>>>>>> data read patterns
>>>>>>>
>>>>>>> cassandra drivers as below:
>>>>>>>
>>>>>>> <dependency>
>>>>>>>     <groupId>com.datastax.cassandra</groupId>
>>>>>>>     <artifactId>cassandra-driver-core</artifactId>
>>>>>>>     <version>2.1.6</version>
>>>>>>> </dependency>
>>>>>>> <dependency>
>>>>>>>     <groupId>com.datastax.cassandra</groupId>
>>>>>>>     <artifactId>cassandra-driver-mapping</artifactId>
>>>>>>>     <version>2.1.6</version>
>>>>>>> </dependency>
>>>>>>>
>>>>>>> cassandra version is 2.0.17
>>>>>>>
>>>>>>> Thanks in advance for the help.
>>>>>>>
>>>>>>> Anishek
>>>>>>>
>>>>>>>
>>>>>
>>>
>

Mime
View raw message