cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From learner dba <cassandra...@yahoo.com.INVALID>
Subject Re: RE: [EXTERNAL] Cluster is unbalanced
Date Mon, 18 Jun 2018 22:07:04 GMT
 Hi Sean,
Are you using any rack aware topology? --> we are using gossip file
 Are you using any rack aware topology? --> we are using gossip file What are your partition
keys? --> Partition key is uniqIs it possible that your partition keys do not divide up
as cleanly as you would like across the cluster because the data is not evenly distributed
(by partition key)?  --> No, we verified it.
If it was partition key issue, we would see similar number of partition keys across nodes.
If we look closely number of keys across nodes vary a lot.

Number of partitions (estimate): 3142552Number of partitions (estimate): 15625442Number of
partitions (estimate): 15244021Number of partitions (estimate): 9592992Number of partitions
(estimate): 15839280




    On Monday, June 18, 2018, 5:39:08 PM EDT, Durity, Sean R <SEAN_R_DURITY@homedepot.com>
wrote:  
 
 #yiv2571733579 #yiv2571733579 -- _filtered #yiv2571733579 {font-family:Helvetica;panose-1:2
11 6 4 2 2 2 2 2 4;} _filtered #yiv2571733579 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv2571733579
{font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv2571733579 #yiv2571733579 p.yiv2571733579MsoNormal,
#yiv2571733579 li.yiv2571733579MsoNormal, #yiv2571733579 div.yiv2571733579MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:12.0pt;font-family:New
serif;}#yiv2571733579 a:link, #yiv2571733579 span.yiv2571733579MsoHyperlink {color:#0563C1;text-decoration:underline;}#yiv2571733579
a:visited, #yiv2571733579 span.yiv2571733579MsoHyperlinkFollowed {color:#954F72;text-decoration:underline;}#yiv2571733579
p {margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:New serif;}#yiv2571733579
p.yiv2571733579msonormal0, #yiv2571733579 li.yiv2571733579msonormal0, #yiv2571733579 div.yiv2571733579msonormal0
{margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:New serif;}#yiv2571733579 p.yiv2571733579ydpe3237830p1,
#yiv2571733579 li.yiv2571733579ydpe3237830p1, #yiv2571733579 div.yiv2571733579ydpe3237830p1
{margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:New serif;}#yiv2571733579 span.yiv2571733579ydpe3237830s1
{}#yiv2571733579 span.yiv2571733579ydpe3237830apple-converted-space {}#yiv2571733579 span.yiv2571733579EmailStyle22
{font-family:sans-serif;color:#1F497D;}#yiv2571733579 .yiv2571733579MsoChpDefault {font-size:10.0pt;}
_filtered #yiv2571733579 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv2571733579 div.yiv2571733579WordSection1
{}#yiv2571733579 
Are you using any rack aware topology? What are your partition keys? Is it possible that your
partition keys do not divide up as cleanly as you would like across the cluster because the
data is not evenly distributed (by partition key)?
 
  
 
  
 
Sean Durity
 
lord of the (C*) rings (Staff Systems Engineer – Cassandra)
 
MTC 2250
 
#cassandra - for the latest news and updates
 
  
 
From: learner dba <cassandradba@yahoo.com.INVALID>
Sent: Monday, June 18, 2018 2:06 PM
To: User cassandra.apache.org <user@cassandra.apache.org>
Subject: [EXTERNAL] Cluster is unbalanced
 
  
 
Hi,
 
  
 
Data volume varies a lot in our two DC cluster:
 
 Load     Tokens     Owns 
 
 20.01 GiB 256         ?     
 
 65.32 GiB 256         ?     
 
 60.09 GiB 256         ?     
 
 46.95 GiB 256         ?     
 
 50.73 GiB 256         ?     
 
kaiprodv2
 
=========
 
/Leaving/Joining/Moving
 
 Load     Tokens     Owns 
 
 25.19 GiB 256         ?     
 
 30.26 GiB 256         ?     
 
 9.82 GiB 256         ?     
 
 20.54 GiB 256         ?     
 
 9.7 GiB    256         ?     
 
  
 
I ran clearsnapshot, garbagecollect and cleanup, but it increased the size on heavier nodes
instead of decreasing. Based on nodetool cfstats, I can see partition keys on each node varies
a lot:
 
  
 
Number of partitions (estimate): 3142552
 
Number of partitions (estimate): 15625442
 
Number of partitions (estimate): 15244021
 
Number of partitions (estimate): 9592992
 
Number of partitions (estimate): 15839280
 
  
 
How can I diagnose this imbalance further?
 
  
   
Mime
View raw message