cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "julia zhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10430) "Load" report from "nodetool status" is inaccurate
Date Thu, 01 Oct 2015 17:52:27 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

julia zhang updated CASSANDRA-10430:
------------------------------------
    Description: 
After running an incremental repair, nodetool status report unbalanced load among cluster.


$ nodetool status mykeyspace
==========================
||Status|| Address         ||Load           ||Tokens  ||Owns (effective)  ||Host ID ||  Rack
||                         
|UN  |10.1.1.1  |1.13 TB       |256    |48.5%            |a4477534-a5c6-4e3e-9108-17a69aebcfc0|
 RAC1|
|UN  |10.1.1.2  |2.58 TB       |256     |50.5%             |1a7c3864-879f-48c5-8dde-bc00cf4b23e6
 |RAC2|
|UN  |10.1.1.3  |1.49 TB       |256     |51.5%             |27df5b30-a5fc-44a5-9a2c-1cd65e1ba3f7
 |RAC1|
|UN  |10.1.1.4  |250.97 GB  |256     |51.9%             |9898a278-2fe6-4da2-b6dc-392e5fda51e6
 |RAC3|
|UN  |10.1.1.5 |1.88 TB      |256     |49.5%             |04aa9ce1-c1c3-4886-8d72-270b024b49b9
 |RAC2|
|UN  |10.1.1.6 |1.3 TB        |256     |48.1%             |6d5d48e6-d188-4f88-808d-dcdbb39fdca5
 |RAC3|

It seems that only 10.1.1.4 reports correct "Load". There is no hints in the cluster and report
remains the same after running "nodetool cleanup" on each node. "nodetool cfstats" shows number
of keys are evenly distributed and Cassandra data physical disk on each node report about
the same usage. 

"nodetool status" report these inaccurate large storage load until we restart each node, after
the restart, "Load" report match what we've seen from disk.  

We did not see this behavior until upgrade to v2.1.9

  was:
After running an incremental repair, nodetool status report unbalanced load among cluster.


$ nodetool status mykeyspace
==========================
||Status|| Address         ||Load           ||Tokens  ||Owns (effective)  ||Host ID ||  Rack
||                         
|UN  |10.6.0.246  |1.13 TB       |256    |48.5%            |a4477534-a5c6-4e3e-9108-17a69aebcfc0|
 RAC1|
|UN  |10.6.0.188  |2.58 TB       |256     |50.5%             |1a7c3864-879f-48c5-8dde-bc00cf4b23e6
 |RAC2|
|UN  |10.6.0.189  |1.49 TB       |256     |51.5%             |27df5b30-a5fc-44a5-9a2c-1cd65e1ba3f7
 |RAC1|
|UN  |10.6.0.186  |250.97 GB  |256     |51.9%             |9898a278-2fe6-4da2-b6dc-392e5fda51e6
 |RAC3|
|UN  |10.6.0.187  |1.88 TB      |256     |49.5%             |04aa9ce1-c1c3-4886-8d72-270b024b49b9
 |RAC2|
|UN  |10.6.0.185  |1.3 TB        |256     |48.1%             |6d5d48e6-d188-4f88-808d-dcdbb39fdca5
 |RAC3|

There is no hints in the cluster and report remains the same after running "nodetool cleanup"
on each node. "nodetool cfstats" shows number of keys are evenly distributed and Cassandra
data physical disk on each node report about the same usage. 

"nodetool status" report this inaccurate large storage load until we restart each node, after
the restart, it reports correct "Load". 

We did not see this behavior until upgrade to v2.1.9


> "Load" report from "nodetool status" is inaccurate
> --------------------------------------------------
>
>                 Key: CASSANDRA-10430
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10430
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>         Environment: Cassandra v2.1.9 running on 6 node Amazon AWS, vnodes enabled. 
>            Reporter: julia zhang
>
> After running an incremental repair, nodetool status report unbalanced load among cluster.

> $ nodetool status mykeyspace
> ==========================
> ||Status|| Address         ||Load           ||Tokens  ||Owns (effective)  ||Host ID ||
 Rack ||                         
> |UN  |10.1.1.1  |1.13 TB       |256    |48.5%            |a4477534-a5c6-4e3e-9108-17a69aebcfc0|
 RAC1|
> |UN  |10.1.1.2  |2.58 TB       |256     |50.5%             |1a7c3864-879f-48c5-8dde-bc00cf4b23e6
 |RAC2|
> |UN  |10.1.1.3  |1.49 TB       |256     |51.5%             |27df5b30-a5fc-44a5-9a2c-1cd65e1ba3f7
 |RAC1|
> |UN  |10.1.1.4  |250.97 GB  |256     |51.9%             |9898a278-2fe6-4da2-b6dc-392e5fda51e6
 |RAC3|
> |UN  |10.1.1.5 |1.88 TB      |256     |49.5%             |04aa9ce1-c1c3-4886-8d72-270b024b49b9
 |RAC2|
> |UN  |10.1.1.6 |1.3 TB        |256     |48.1%             |6d5d48e6-d188-4f88-808d-dcdbb39fdca5
 |RAC3|
> It seems that only 10.1.1.4 reports correct "Load". There is no hints in the cluster
and report remains the same after running "nodetool cleanup" on each node. "nodetool cfstats"
shows number of keys are evenly distributed and Cassandra data physical disk on each node
report about the same usage. 
> "nodetool status" report these inaccurate large storage load until we restart each node,
after the restart, "Load" report match what we've seen from disk.  
> We did not see this behavior until upgrade to v2.1.9



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message