incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkat Rama <venkata.s.r...@gmail.com>
Subject Re: Monitoring replication lag/latency in multi DC setup
Date Thu, 06 Sep 2012 05:33:12 GMT
Is there a specific metric you can recommend?

VR

On Wed, Sep 5, 2012 at 9:19 PM, Mohit Anchlia <mohitanchlia@gmail.com>wrote:

> Cassandra exposes lot of metrics through Jconsole. You might be able to
> get some information from Jconsole.
>
>
> On Wed, Sep 5, 2012 at 8:47 PM, Venkat Rama <venkata.s.rama@gmail.com>wrote:
>
>> Thanks for the quick reply, Mohit.    Can we measure/monitor the size of
>> Hinted Handoffs?  Would it be a good enough indicator of my back log?
>>
>> Although we know when a network is flaky, we are interested in knowing
>> how much data is piling up in local DC that needs to be transferred.
>>
>> Greatly appreciate your help.
>>
>> VR
>>
>>
>> On Wed, Sep 5, 2012 at 8:33 PM, Mohit Anchlia <mohitanchlia@gmail.com>wrote:
>>
>>> As far as I know Cassandra doesn't use internal queueing mechanism
>>> specific to replication. Cassandra sends the write the remote DC and after
>>> that it's upto the tcp/ip stack to deal with buffering. If requests starts
>>> to timeout Cassandra would use HH upto certain time. For longer outage you
>>> would have to run repair.
>>>
>>> Also look at tcp/ip tuning parameters that are helpful with your
>>> scenario:
>>>
>>> http://kaivanov.blogspot.com/2010/09/linux-tcp-tuning.html
>>>
>>> Run iperf and test the latency.
>>>
>>>  On Wed, Sep 5, 2012 at 8:22 PM, Venkat Rama <venkata.s.rama@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> We have multi DC Cassandra ring with 2 DCs setup.   We use LOCAL_QUORUM
>>>> for writes and reads.  The network we have seen between the DC is sometimes
>>>> flaky lasting few minutes to few 10 of minutes.
>>>>
>>>> I wanted to know what is the best way to measure/monitor either the lag
>>>> or replication latency between the data centers.  Are there any metrics I
>>>> can monitor to find the backlog of data that needs to be transferred?
>>>>
>>>> Thanks in advance.
>>>>
>>>> VR
>>>>
>>>
>>>
>>
>

Mime
View raw message