Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of ben@instaclustr.com
 designates 209.85.220.48 as permitted sender)
From: Ben Bromhead <ben@instaclustr.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_6CB36475-501A-4954-98FD-B3599ECCD6DB"
Message-Id: <AB6CAF84-924B-46B1-9EED-53C10F8D378A@instaclustr.com>
Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\))
Subject: Re: Multi-DC Environment Question
Date: Fri, 30 May 2014 12:13:00 +1000
References: <5387B31E.7030206@gmail.com>
 <CADjRqh_HBh+wQw17W3hOMDN8U8gwhFORETWb0FXtcvgD1B1o-Q@mail.gmail.com>
 <CADjRqh_1ZChNuMAm0DMY3PiERtfJe9B6FsQ9DEO-Z1ZyUYS5Gw@mail.gmail.com>
 <CADjRqh_joHAHY4Z7cN27F5rYknZtM8LCZ4s31Trz=sJLfkP2mg@mail.gmail.com>
To: user@cassandra.apache.org
In-Reply-To: 
 <CADjRqh_joHAHY4Z7cN27F5rYknZtM8LCZ4s31Trz=sJLfkP2mg@mail.gmail.com>


--Apple-Mail=_6CB36475-501A-4954-98FD-B3599ECCD6DB
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Short answer:

If time elapsed > max_hint_window_in_ms then hints will stop being =
created. You will need to rely on your read consistency level, read =
repair and anti-entropy repair operations to restore consistency.

Long answer:

=
http://www.slideshare.net/jasedbrown/understanding-antientropy-in-cassandr=
a

Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359

On 30 May 2014, at 8:40 am, Tupshin Harper <tupshin@tupshin.com> wrote:

> When one node or DC is down, coordinator nodes being written through =
will notice this fact and store hints (hinted handoff is the mechanism), =
 and those hints are used to send the data that was not able to be =
replicated initially.
>=20
> http://www.datastax.com/dev/blog/modern-hinted-handoff
>=20
> -Tupshin
>=20
> On May 29, 2014 6:22 PM, "Vasileios Vlachos" =
<vasileiosvlachos@gmail.com> wrote:
> Hello All,
>=20
> We have plans to add a second DC to our live Cassandra environment. =
Currently RF=3D3 and we read and write at QUORUM. After adding DC2 we =
are going to be reading and writing at LOCAL_QUORUM.
>=20
> If my understanding is correct, when a client sends a write request, =
if the consistency level is satisfied on DC1 (that is RF/2+1), success =
is returned to the client and DC2 will eventually get the data as well. =
The assumption behind this is that the the client always connects to DC1 =
for reads and writes and given that there is a site-to-site VPN between =
DC1 and DC2. Therefore, DC1 will almost always return success before DC2 =
(actually I don't know if it is possible for DC2 to be more up-to-date =
than DC1 with this setup...).
>=20
> Now imagine DC1 looses connectivity and the client fails over to DC2. =
Everything should work fine after that, with the only difference that =
DC2 will be now handling the requests directly from the client. After =
some time, say after max_hint_window_in_ms, DC1 comes back up. My =
question is how do I bring DC1 up to speed with DC2 which is now more =
up-to-date? Will that require a nodetool repair on DC1 nodes? Also, what =
is the answer when the outage is < max_hint_window_in_ms instead?
>=20
> Thanks in advance!
>=20
> Vasilis
> --=20
> Kind Regards,
>=20
> Vasileios Vlachos


--Apple-Mail=_6CB36475-501A-4954-98FD-B3599ECCD6DB
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Short answer:</div><div><br></div><div>If time elapsed &gt;&nbsp;<span style="font-family: 'Courier New', Courier, monospace;">max_hint_window_in_ms</span>&nbsp;then hints will stop being created. You will need to rely on your read consistency level, read repair and anti-entropy repair operations to restore consistency.</div><div><br></div><div>Long answer:</div><div><br></div><a href="http://www.slideshare.net/jasedbrown/understanding-antientropy-in-cassandra">http://www.slideshare.net/jasedbrown/understanding-antientropy-in-cassandra</a><div><br></div><div><div>
<div><div><div>Ben Bromhead</div><div></div></div><div>Instaclustr |&nbsp;<a href="https://www.instaclustr.com/">www.instaclustr.com</a>&nbsp;|&nbsp;<a href="http://twitter.com/instaclustr">@instaclustr</a>&nbsp;| +61 415 936 359</div></div>

</div>
<br><div><div>On 30 May 2014, at 8:40 am, Tupshin Harper &lt;<a href="mailto:tupshin@tupshin.com">tupshin@tupshin.com</a>&gt; wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><p dir="ltr">When one node or DC is down, coordinator nodes being written through will notice this fact and store hints (hinted handoff is the mechanism),&nbsp; and those hints are used to send the data that was not able to be replicated initially. </p><p dir="ltr"><a href="http://www.datastax.com/dev/blog/modern-hinted-handoff">http://www.datastax.com/dev/blog/modern-hinted-handoff</a></p><p dir="ltr">-Tupshin </p>
<div class="gmail_quote">On May 29, 2014 6:22 PM, "Vasileios Vlachos" &lt;<a href="mailto:vasileiosvlachos@gmail.com">vasileiosvlachos@gmail.com</a>&gt; wrote:<br type="attribution"><blockquote class="quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; position: static; z-index: auto;">

  
  <div bgcolor="#CCCCFF" text="#000000">
    <font size="-1">Hello All,<br>
      <br>
    </font><small>We have plans to add a second DC to our live Cassandra
      environment. Currently RF=3 and we read and write at QUORUM. After
      adding DC2 we are going to be reading and writing at LOCAL_QUORUM.<br>
      <br>
      If my understanding is correct, when a client sends a write
      request, if the consistency level is satisfied on DC1 (that is
      RF/2+1), success is returned to the client and DC2 will eventually
      get the data as well. The assumption behind this is that the the
      client always connects to DC1 for reads and writes and given that
      there is a site-to-site VPN between DC1 and DC2. Therefore, DC1
      will almost always return success before DC2 (actually I don't
      know if it is possible for DC2 to be more up-to-date than DC1 with
      this setup...).<br>
      <br>
      Now imagine DC1 looses connectivity and the client fails over to
      DC2. Everything should work fine after that, with the only
      difference that DC2 will be now handling the requests directly
      from the client. After some time, say after <font face="Courier
        New, Courier, monospace">max_hint_window_in_ms</font>, DC1 comes
      back up. My question is how do I bring DC1 up to speed with DC2
      which is now more up-to-date? Will that require a <font face="Courier New, Courier, monospace">nodetool repair</font> on
      DC1 nodes? Also, what is the answer when the outage is &lt; </small><small><font face="Courier New, Courier, monospace">max_hint_window_in_ms</font><small>
        <big>instead?<br>
          <br>
          Thanks in advance!<br>
          <br>
          Vasilis</big></small></small><font color="#888888"><br>
    <pre cols="72">-- 
Kind Regards,

Vasileios Vlachos</pre>
  </font></div>

</blockquote></div>
</blockquote></div><br></div></body></html>
--Apple-Mail=_6CB36475-501A-4954-98FD-B3599ECCD6DB--