incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Multi datacenter, WAN hiccups and replication
Date Wed, 27 Jun 2012 11:39:26 GMT
> Therefore I was wondering if Cassandra already intelligently optimizes
> for HH-over-WAN (since this is common) or alternately if there's a way
> to enable HH for WAN replication?
When the coordinator is preparing to process the request, done nodes in a foreign DC are treated
like down nodes in a local DC. So a hint is stored for each down node. So more than 1 cross
DC message is used when the hints are replayed. 

When the local DC coordinator sends a forwarded message to a foreign DC coordinator it sets
up an expectation to hear back from each of the replicas in the foreign DC. If any of these
fail to return, because say the foreign DC coordinator went down, a hint is stored. So again,
more than 1 cross DC message will be used when the hints are replayed. 

Replaying hints is triggered when a node notices that another is up again. There would be
no sense in forwarding the messages as they are sent directly to the recipient.  

Hope that helps. 
   
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 27/06/2012, at 5:14 AM, Karthik N wrote:

> Let me attempt to articulate my question a little better.
> 
> Say I choose LOCAL_QUORUM with a Replication Factor of 3. Cassandra
> stores three copies in my local datacenter. Therefore the cost
> associated with "losing" one node is not very high locally, and I
> usually HH, and use read repair/nodetool repair instead.
> 
> However over the WAN network blips are quite normal and HH really
> helps. More so because for WAN replication Cassandra sends only one
> copy to a coordinator in the remote datacenter.
> 
> Therefore I was wondering if Cassandra already intelligently optimizes
> for HH-over-WAN (since this is common) or alternately if there's a way
> to enable HH for WAN replication?
> 
> Thank you.
> 
> On Tue, Jun 26, 2012 at 9:22 AM, Mohit Anchlia <mohitanchlia@gmail.com> wrote:
>> 
>> 
>> On Tue, Jun 26, 2012 at 8:16 AM, Karthik N <karthik.nar@gmail.com> wrote:
>>> 
>>> Since Cassandra optimizes and sends only one copy over the WAN, can I opt
>>> in only for HH for WAN replication and avoid HH for the local quorum? (since
>>> I know I have more copies)
>>> 
>>> 
>> 
>> I am not sure if I understand your question. In general I don't think you
>> can selectively decide on HH. Besides HH should only be used when the outage
>> is in mts, for longer outages using HH would only create memory pressure.
>>> 
>>> On Tuesday, June 26, 2012, Mohit Anchlia wrote:
>>>> 
>>>> 
>>>> On Tue, Jun 26, 2012 at 7:52 AM, Karthik N <karthik.nar@gmail.com>
wrote:
>>>>> 
>>>>> My Cassandra ring spans two DCs. I use local quorum with replication
>>>>> factor=3. I do a write in DC1 with local quorum. Data gets written to
>>>>> multiple nodes in DC1. For the same write to propagate to DC2 only one
>>>>> copy is sent from the coordinator node in DC1 to a coordinator node in
>>>>> DC2 for optimizing traffic over the WAN (from what I have read in the
>>>>> Cassandra documentation)
>>>>> 
>>>>> Will a Wan hiccup result in a Hinted Handoff (HH) being created in
>>>>> DC1's coordinator for DC2 to be delivered when the Wan link is up
>>>>> again?
>>>> 
>>>> 
>>>> I have seen hinted handoff messages in the log files when the remote DC
>>>> is unreachable. But this mechanism is only used for a the time defined in
>>>> cassandra.yaml file.
>>> 
>>> 
>>> 
>>> --
>>> Thanks, Karthik
>> 
>> 


Mime
View raw message