incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vasileios Vlachos <>
Subject Re: Multi-DC Environment Question
Date Mon, 16 Jun 2014 08:32:21 GMT
Hello again,

Back to this after a while...

As far as I can tell whenever DC2 is unavailable, there is one node from
DC1 that acts as a coordinator. When DC2 is available again, this one node
sends the hints to only one node at DC2, which then sends any replicas to
the other nodes in the local DC (DC2). This ensures efficient cross-DC
bandwidth usage. I was watching "system.hints" on all nodes during this
test and this is the conclusion I came to.

Two things:
1. If the above is correct, does the same apply when performing
anti-entropy repair (without specifying a particular DC)? I'm just hoping
the answer to this is going to be YES, otherwise the VPN is not going to be
very happy in our case and we would prefer to not saturate it whenever
running nodetool repair. I suppose we could have a traffic limiter on the
firewalls worst case scenario but I would appreciate your input if you know
more on this.

2. As I described earlier, in order to test this I was watching the
"system.hints" CF in order to monitor any hints. I was looking to add a
Nagios check for this purpose. For that reason I was looking into JMX
Concole. I noticed that when a node stores hints, "MBean
attribute "MemtableColumnsCount" goes up (although I would expect it to be
MemtableRowCount or something?). This attribute will retain its value,
until the other node becomes available and ready to receive the hints. I
was looking for another attribute somewhere to monitor the active hints. I


"MBean org.apache.cassandra.metrics:type=Storage,name=TotalHints",
and even
org.apache.cassandra.metrics:type=HintedHandOffManager,name=Hints_not_stored-/" (this one will never go back to zero).

All of them would not increase whenever any hints are being sent (or at
least I didn't catch it because it was too fast or whatever?). Does anyone
know what all these attributes represent? It looks like there are more
specific hint attributes on a per CF basis, but I was looking for a more
generic one to begin with. Any help would be much appreciated.

Thanks in advance,


On Wed, Jun 4, 2014 at 1:42 PM, Vasileios Vlachos <> wrote:

> Hello Matt,
> nodetool status:
> Datacenter: MAN
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Owns (effective) Host ID Token Rack
> UN 89.34 KB 99.2% b7f8bc93-bf39-475c-a251-8fbe2c7f7239
> -9211685935328163899 RAC1
> UN 86.32 KB 0.7% 1f8937e1-9ecb-4e59-896e-6d6ac42dc16d
> -3511707179720619260 RAC1
> Datacenter: DER
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Owns (effective) Host ID Token Rack
> UN 75.43 KB 0.2% e71c7ee7-d852-4819-81c0-e993ca87dd5c
> -1277931707251349874 RAC1
> UN 104.53 KB 99.8% 7333b664-ce2d-40cf-986f-d4b4d4023726
> -9204412570946850701 RAC1
> I do not know why the cluster is not balanced at the moment, but it holds
> almost no data. I will populate it soon and see how that goes. The output
> of 'nodetool ring' just lists all the tokens assigned to each individual
> node, and as you can imagine it would be pointless to paste it here. I just
> did 'nodetool ring | awk ... | unique | wc -l' and it works out to be 1024
> as expected (4 nodes x 256 tokens each).
> Still have not got the answers to the other questions though...
> Thanks,
> Vasilis
> On Wed, Jun 4, 2014 at 12:28 AM, Matthew Allen <>
> wrote:
>> Thanks Vasileios.  I think I need to make a call as to whether to switch
>> to vnodes or stick with tokens for my Multi-DC cluster.
>> Would you be able to show a nodetool ring/status from your cluster to see
>> what the token assignment looks like ?
>> Thanks
>> Matt
>> On Wed, Jun 4, 2014 at 8:31 AM, Vasileios Vlachos <
>>> wrote:
>>>  I should have said that earlier really... I am using 1.2.16 and Vnodes
>>> are enabled.
>>> Thanks,
>>> Vasilis
>>> --
>>> Kind Regards,
>>> Vasileios Vlachos

View raw message