Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <1490862146.8615.4.camel@willhaben.at>
References: <1490862146.8615.4.camel@willhaben.at>
From: Chris Lohfink <clohfink85@gmail.com>
Date: Sat, 1 Apr 2017 12:14:08 -0700
Message-ID: <CAOsmgAwhjWQhy57C2hm=yWgaiX7HRcbQX8q-KoesRp34=o_s8g@mail.gmail.com>
Subject: Re: nodes are always out of sync
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=94eb2c04448e004bee054c1fbbf5
archived-at: Sat, 01 Apr 2017 19:14:18 -0000

--94eb2c04448e004bee054c1fbbf5
Content-Type: text/plain; charset=UTF-8

Repairs do not have an ability to instantly build a perfect view of its
data between your 3 nodes at an exact time. When a piece of data is written
there is a delay between when they applied between the nodes, even if its
just 500ms. So if a request to read the data and build the merkle tree of
the data occurs and it finishes on node1 at 12:01 while node2 finishes at
12:02 the 1 minute or so delta (even if a few seconds, or if using snapshot
repairs) between the partition/range hashes in the merkle tree can be
different. On a moving data set its almost impossible to have the clusters
perfectly in sync for a repair. I wouldnt worry about that log message. If
you are worried about consistency between your read/writes use each or
local quorum for both.

Chris

On Thu, Mar 30, 2017 at 1:22 AM, Roland Otta <Roland.Otta@willhaben.at>
wrote:

> hi,
>
> we see the following behaviour in our environment:
>
> cluster consists of 6 nodes (cassandra version 3.0.7). keyspace has a
> replication factor 3.
> clients are writing data to the keyspace with consistency one.
>
> we are doing parallel, incremental repairs with cassandra reaper.
>
> even if a repair just finished and we are starting a new one
> immediately, we can see the following entries in our logs:
>
> INFO  [RepairJobTask:1] 2017-03-30 10:14:00,782 SyncTask.java:73 -
> [repair #d0f651f6-1520-11e7-a443-d9f5b942818e] Endpoints /192.168.0.188
> and /192.168.0.191 have 1 range(s) out of sync for ad_event_history
> INFO  [RepairJobTask:2] 2017-03-30 10:14:00,782 SyncTask.java:73 -
> [repair #d0f651f6-1520-11e7-a443-d9f5b942818e] Endpoints /192.168.0.188
> and /192.168.0.189 have 1 range(s) out of sync for ad_event_history
> INFO  [RepairJobTask:4] 2017-03-30 10:14:00,782 SyncTask.java:73 -
> [repair #d0f651f6-1520-11e7-a443-d9f5b942818e] Endpoints /192.168.0.189
> and /192.168.0.191 have 1 range(s) out of sync for ad_event_history
> INFO  [RepairJobTask:2] 2017-03-30 10:14:03,997 SyncTask.java:73 -
> [repair #d0fa70a1-1520-11e7-a443-d9f5b942818e] Endpoints /192.168.0.26
> and /192.168.0.189 have 2 range(s) out of sync for ad_event_history
> INFO  [RepairJobTask:1] 2017-03-30 10:14:03,997 SyncTask.java:73 -
> [repair #d0fa70a1-1520-11e7-a443-d9f5b942818e] Endpoints /192.168.0.26
> and /192.168.0.191 have 2 range(s) out of sync for ad_event_history
> INFO  [RepairJobTask:4] 2017-03-30 10:14:03,997 SyncTask.java:73 -
> [repair #d0fa70a1-1520-11e7-a443-d9f5b942818e] Endpoints /192.168.0.189
> and /192.168.0.191 have 2 range(s) out of sync for ad_event_history
> INFO  [RepairJobTask:1] 2017-03-30 10:14:05,375 SyncTask.java:73 -
> [repair #d0fbd033-1520-11e7-a443-d9f5b942818e] Endpoints /192.168.0.189
> and /192.168.0.191 have 1 range(s) out of sync for ad_event_history
> INFO  [RepairJobTask:2] 2017-03-30 10:14:05,375 SyncTask.java:73 -
> [repair #d0fbd033-1520-11e7-a443-d9f5b942818e] Endpoints /192.168.0.189
> and /192.168.0.190 have 1 range(s) out of sync for ad_event_history
> INFO  [RepairJobTask:4] 2017-03-30 10:14:05,375 SyncTask.java:73 -
> [repair #d0fbd033-1520-11e7-a443-d9f5b942818e] Endpoints /192.168.0.190
> and /192.168.0.191 have 1 range(s) out of sync for ad_event_history
>
> we cant see any hints on the systems ... so we thought everything is
> running smoothly with the writes.
>
> do we have to be concerned about the nodes always being out of sync or
> is this a normal behaviour in a write intensive table (as the tables
> will never be 100% in sync for the latest inserts)?
>
> bg,
> roland
>
>
>

--94eb2c04448e004bee054c1fbbf5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Repairs do not have an ability to instantly build a perfec=
t view of its data between your 3 nodes at an exact time. When a piece of d=
ata is written there is a delay between when they applied between the nodes=
, even if its just 500ms. So if a request to read the data and build the me=
rkle tree of the data occurs and it finishes on node1 at 12:01 while node2 =
finishes at 12:02 the 1 minute or so delta (even if a few seconds, or if us=
ing snapshot repairs) between the partition/range hashes in the merkle tree=
 can be different. On a moving data set its almost impossible to have the c=
lusters perfectly in sync for a repair. I wouldnt worry about that log mess=
age. If you are worried about consistency between your read/writes use each=
 or local quorum for both.<div><br></div><div>Chris</div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, Mar 30, 2017 at 1:2=
2 AM, Roland Otta <span dir=3D"ltr">&lt;<a href=3D"mailto:Roland.Otta@willh=
aben.at" target=3D"_blank">Roland.Otta@willhaben.at</a>&gt;</span> wrote:<b=
r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:=
1px #ccc solid;padding-left:1ex">hi,<br>
<br>
we see the following behaviour in our environment:<br>
<br>
cluster consists of 6 nodes (cassandra version 3.0.7). keyspace has a<br>
replication factor 3.<br>
clients are writing data to the keyspace with consistency one.<br>
<br>
we are doing parallel, incremental repairs with cassandra reaper.<br>
<br>
even if a repair just finished and we are starting a new one<br>
immediately, we can see the following entries in our logs:<br>
<br>
INFO=C2=A0=C2=A0[RepairJobTask:1] 2017-03-30 10:14:00,782 SyncTask.java:73 =
-<br>
[repair #d0f651f6-1520-11e7-a443-<wbr>d9f5b942818e] Endpoints /<a href=3D"h=
ttp://192.168.0.188" rel=3D"noreferrer" target=3D"_blank">192.168.0.188</a>=
<br>
and /<a href=3D"http://192.168.0.191" rel=3D"noreferrer" target=3D"_blank">=
192.168.0.191</a> have 1 range(s) out of sync for ad_event_history<br>
INFO=C2=A0=C2=A0[RepairJobTask:2] 2017-03-30 10:14:00,782 SyncTask.java:73 =
-<br>
[repair #d0f651f6-1520-11e7-a443-<wbr>d9f5b942818e] Endpoints /<a href=3D"h=
ttp://192.168.0.188" rel=3D"noreferrer" target=3D"_blank">192.168.0.188</a>=
<br>
and /<a href=3D"http://192.168.0.189" rel=3D"noreferrer" target=3D"_blank">=
192.168.0.189</a> have 1 range(s) out of sync for ad_event_history<br>
INFO=C2=A0=C2=A0[RepairJobTask:4] 2017-03-30 10:14:00,782 SyncTask.java:73 =
-<br>
[repair #d0f651f6-1520-11e7-a443-<wbr>d9f5b942818e] Endpoints /<a href=3D"h=
ttp://192.168.0.189" rel=3D"noreferrer" target=3D"_blank">192.168.0.189</a>=
<br>
and /<a href=3D"http://192.168.0.191" rel=3D"noreferrer" target=3D"_blank">=
192.168.0.191</a> have 1 range(s) out of sync for ad_event_history<br>
INFO=C2=A0=C2=A0[RepairJobTask:2] 2017-03-30 10:14:03,997 SyncTask.java:73 =
-<br>
[repair #d0fa70a1-1520-11e7-a443-<wbr>d9f5b942818e] Endpoints /<a href=3D"h=
ttp://192.168.0.26" rel=3D"noreferrer" target=3D"_blank">192.168.0.26</a><b=
r>
and /<a href=3D"http://192.168.0.189" rel=3D"noreferrer" target=3D"_blank">=
192.168.0.189</a> have 2 range(s) out of sync for ad_event_history<br>
INFO=C2=A0=C2=A0[RepairJobTask:1] 2017-03-30 10:14:03,997 SyncTask.java:73 =
-<br>
[repair #d0fa70a1-1520-11e7-a443-<wbr>d9f5b942818e] Endpoints /<a href=3D"h=
ttp://192.168.0.26" rel=3D"noreferrer" target=3D"_blank">192.168.0.26</a><b=
r>
and /<a href=3D"http://192.168.0.191" rel=3D"noreferrer" target=3D"_blank">=
192.168.0.191</a> have 2 range(s) out of sync for ad_event_history<br>
INFO=C2=A0=C2=A0[RepairJobTask:4] 2017-03-30 10:14:03,997 SyncTask.java:73 =
-<br>
[repair #d0fa70a1-1520-11e7-a443-<wbr>d9f5b942818e] Endpoints /<a href=3D"h=
ttp://192.168.0.189" rel=3D"noreferrer" target=3D"_blank">192.168.0.189</a>=
<br>
and /<a href=3D"http://192.168.0.191" rel=3D"noreferrer" target=3D"_blank">=
192.168.0.191</a> have 2 range(s) out of sync for ad_event_history<br>
INFO=C2=A0=C2=A0[RepairJobTask:1] 2017-03-30 10:14:05,375 SyncTask.java:73 =
-<br>
[repair #d0fbd033-1520-11e7-a443-<wbr>d9f5b942818e] Endpoints /<a href=3D"h=
ttp://192.168.0.189" rel=3D"noreferrer" target=3D"_blank">192.168.0.189</a>=
<br>
and /<a href=3D"http://192.168.0.191" rel=3D"noreferrer" target=3D"_blank">=
192.168.0.191</a> have 1 range(s) out of sync for ad_event_history<br>
INFO=C2=A0=C2=A0[RepairJobTask:2] 2017-03-30 10:14:05,375 SyncTask.java:73 =
-<br>
[repair #d0fbd033-1520-11e7-a443-<wbr>d9f5b942818e] Endpoints /<a href=3D"h=
ttp://192.168.0.189" rel=3D"noreferrer" target=3D"_blank">192.168.0.189</a>=
<br>
and /<a href=3D"http://192.168.0.190" rel=3D"noreferrer" target=3D"_blank">=
192.168.0.190</a> have 1 range(s) out of sync for ad_event_history<br>
INFO=C2=A0=C2=A0[RepairJobTask:4] 2017-03-30 10:14:05,375 SyncTask.java:73 =
-<br>
[repair #d0fbd033-1520-11e7-a443-<wbr>d9f5b942818e] Endpoints /<a href=3D"h=
ttp://192.168.0.190" rel=3D"noreferrer" target=3D"_blank">192.168.0.190</a>=
<br>
and /<a href=3D"http://192.168.0.191" rel=3D"noreferrer" target=3D"_blank">=
192.168.0.191</a> have 1 range(s) out of sync for ad_event_history<br>
<br>
we cant see any hints on the systems ... so we thought everything is<br>
running smoothly with the writes.<br>
<br>
do we have to be concerned about the nodes always being out of sync or<br>
is this a normal behaviour in a write intensive table (as the tables<br>
will never be 100% in sync for the latest inserts)?<br>
<br>
bg,<br>
roland<br>
<br>
<br>
</blockquote></div><br></div>

--94eb2c04448e004bee054c1fbbf5--