incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Hobbs <ty...@datastax.com>
Subject Re: new "nodetool ring" output and unbalanced ring?
Date Thu, 06 Sep 2012 18:09:20 GMT
The main issue is that one of your us-east nodes is in rack 1d, while the
restart are in rack 1c.  With NTS and multiple racks, Cassandra will try
use one node from each rack as a replica for a range until it either meets
the RF for the DC, or runs out of racks, in which case it just picks nodes
sequentially going clockwise around the ring (starting from the range being
considered, not the last node that was chosen as a replica).

To fix this, you'll either need to make the 1d node a 1c node, or make
42535295865117307932921825928971026432 a 1d node so that you're alternating
racks within that DC.

On Thu, Sep 6, 2012 at 12:54 PM, William Oberman
<oberman@civicscience.com>wrote:

> Hi,
>
> I recently upgraded from 0.8.x to 1.1.x (through 1.0 briefly) and nodetool
> -ring seems to have changed from "owns" to "effectively owns".
>  "Effectively owns" seems to account for replication factor (RF).  I'm ok
> with all of this, yet I still can't figure out what's up with my cluster.
>  I have a NetworkTopologyStrategy with two data centers (DCs) with
> RF/number nodes in DC combinations of:
> DC Name, RF, # in DC
> analytics, 1, 2
> us-east, 3, 4
> So I'd expect 50% on each analytics node, and 75% for each us-east node.
>  Instead, I have two nodes in us-east with 50/100??? (the other two are
> 75/75 as expected).
>
> Here is the output of nodetool (all nodes report the same thing):
> Address         DC          Rack        Status State   Load
>  Effective-Ownership Token
>
>                  127605887595351923798765477786913079296
> x.x.x.x   us-east     1c          Up     Normal  94.57 GB        75.00%
>            0
> x.x.x.x   analytics   1c          Up     Normal  60.64 GB        50.00%
>            1
> x.x.x.x   us-east     1c          Up     Normal  131.76 GB       75.00%
>            42535295865117307932921825928971026432
> x.x.x.x    us-east     1c          Up     Normal  43.45 GB        50.00%
>            85070591730234615865843651857942052864
> x.x.x.x    analytics   1d          Up     Normal  60.88 GB        50.00%
>            85070591730234615865843651857942052865
> x.x.x.x   us-east     1d          Up     Normal  98.56 GB        100.00%
>           127605887595351923798765477786913079296
>
> If I use cassandra-cli to do "show keyspaces;" I get (and again, all nodes
> report the same thing):
> Keyspace: civicscience:
>   Replication Strategy:
> org.apache.cassandra.locator.NetworkTopologyStrategy
>   Durable Writes: true
>     Options: [analytics:1, us-east:3]
> I removed the output about all of my column families (CFs), hopefully that
> doesn't matter.
>
> Did I compute the tokens wrong?  Is there a combination of nodetool
> commands I can run to migrate the data around to rebalance to 75/75/75/75?
>  I routinely run repair already.  And as the release notes required, I ran
> upgradesstables during the upgrade process.
>
> Before the upgrade, I was getting analytics = 0%, and us-east = 25% on
> each node, which I expected for "owns".
>
> will
>
>


-- 
Tyler Hobbs
DataStax <http://datastax.com/>

Mime
View raw message