Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <1893276950.6705564.1474633255633@mail.yahoo.com>
References: <CABF4AKg1KEJRWrWUm+_baBv+O7=w9xxTrU8z8tuuju+UV-DzCQ@mail.gmail.com>
 <CAL+ArfXuwseAgKPGScCr6g4odoK_0Pe6-d7pfZP7u_nekmkHOg@mail.gmail.com>
 <CA+VSrLrMZG2b9U+_2FB79jUVC+CVN4DRo89E8B4EL+T8ZBjMdw@mail.gmail.com>
 <CAL+ArfV9k_UrMfMNy1=v4aNFV4QkG6RJGLC_Xq-RxRL2q51Dhw@mail.gmail.com>
 <CABZU2BHH3qHBdYfqmCp-kKBoRbNR8b8Heo3QPJQryszUY_dz4A@mail.gmail.com>
 <CA+VSrLp7HDvbb0t9KZNbAzo3orQGk6NaUOGKK4=qq2XnJbNfkw@mail.gmail.com>
 <CABZU2BGWUfxcZ3KEStkKDAompvB_bKx2UmoaKr8NCRO1_sheUQ@mail.gmail.com>
 <1061400395.4655637.1474473114779@mail.yahoo.com> <CABZU2BGKRK1Kp342Ua5=GPS=jTGponfJcdC9Uajge3JraDKaGA@mail.gmail.com>
 <2132965820.5615001.1474546996017@mail.yahoo.com> <CABZU2BHGpyDfk_63gcv1NXMio3uGjSiRJ3cic+GiTtWQd7LD0g@mail.gmail.com>
 <1513663612.5903714.1474558986196@mail.yahoo.com> <953704496.5874903.1474559274477@mail.yahoo.com>
 <CABZU2BHnoqZsNhBa6+3jp9Ec-+py28dOAi-OB5WDpgF_qtdDrw@mail.gmail.com> <1893276950.6705564.1474633255633@mail.yahoo.com>
From: "Li, Guangxing" <guangxing.li@pearson.com>
Date: Thu, 29 Sep 2016 11:19:30 -0600
Message-ID: <CABZU2BE-T8KHs73U1wn0zbZEKtgyUjc-cx7_V+Cw8UTtrUZwJw@mail.gmail.com>
Subject: Re: Nodetool repair
To: user@cassandra.apache.org, Romain Hardouin <romainh_ml@yahoo.fr>
Content-Type: multipart/alternative; boundary=001a1145f7ce46e24f053da8ae36
archived-at: Thu, 29 Sep 2016 17:20:12 -0000

--001a1145f7ce46e24f053da8ae36
Content-Type: text/plain; charset=UTF-8

Romain,

I was trying what you mentioned as below:

a. nodetool stop VALIDATION
b. echo run -b org.apache.cassandra.db:type=StorageService
forceTerminateAllRepairSessions | java -jar
/tmp/jmxterm/jmxterm-1.0-alpha-4-uber.jar
-l 127.0.0.1:7199

to stop a seemingly forever-going repair but seeing really odd behavior
with C* 2.0.9. Here is what I did:
1. First, I run 'nodetool tpstats' on all nodes in the cluster and seeing
only one node have 1 active pending AntiEntropySessions. All other nodes do
not have any pending or active AntiEntropySessions.
2. Then I grep 'Repair' on all logs on all nodes and seeing absolutely no
repair related activity in these logs for the past day.
3. Then on the node that has active AntiEntropySessions, I did steps 'a'
and 'b' above. Now all the sudden I start seeing repair activities, on
nodes that did not have pending AntiEntropySessions, I am seeing the
following in their logs:
INFO [NonPeriodicTasks:1] 2016-09-29 17:12:53,469 StreamingRepairTask.java
(line 87) [repair #e80e17d0-8667-11e6-a801-e172d7a67134] streaming task
succeed, returning response to /10.253.2.166
On node 10.253.2.166 which has active pending AntiEntropySessions, I am
seeing the following in the log:
INFO [AntiEntropySessions:136] 2016-09-29 17:03:02,405 RepairSession.java
(line 282) [repair #812dafe0-8666-11e6-a801-e172d7a67134] session completed
successfully

So it seems to me that by doing forceTerminateAllRepairSessions, it
actually 'wakes up' the dormant repair so it goes again. So far, the only
way I can get working to stop a repair is to restart C* node where the
repair command is initiated.

Thanks.

George.

On Fri, Sep 23, 2016 at 6:20 AM, Romain Hardouin <romainh_ml@yahoo.fr>
wrote:

> OK. If you still have issues after setting streaming_socket_timeout_in_ms
> != 0, consider increasing request_timeout_in_ms to a high value, say 1 or 2
> minutes. See comments in https://issues.apache.org/
> jira/browse/CASSANDRA-7904
> Regarding 2.1, be sure to test incremental repair on your data before to
> run it in production ;-)
>
> Romain
>

--001a1145f7ce46e24f053da8ae36
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><font face=3D"arial, helvetica, sans-serif">Romain,</font>=
<div><font face=3D"arial, helvetica, sans-serif"><br></font></div><div><fon=
t face=3D"arial, helvetica, sans-serif">I was trying what you mentioned as =
below:</font></div><div><font face=3D"arial, helvetica, sans-serif"><br></f=
ont></div><div><div dir=3D"ltr" style=3D"color:rgb(0,0,0)"><font face=3D"ar=
ial, helvetica, sans-serif">a.=C2=A0<font size=3D"2">nodetool stop VALIDATI=
ON</font></font></div><div dir=3D"ltr" style=3D"color:rgb(0,0,0)"><font siz=
e=3D"2" face=3D"arial, helvetica, sans-serif">b. echo run -b org.apache.cas=
sandra.db:type=3D<wbr>StorageService forceTerminateAllRepairSession<wbr>s |=
 java -jar /tmp/jmxterm/jmxterm-1.0-<wbr>alpha-4-uber.jar -l=C2=A0<a href=
=3D"http://127.0.0.1:7199/" target=3D"_blank">127.0.0.1:7199</a></font></di=
v></div><div style=3D"color:rgb(0,0,0)"><font face=3D"arial, helvetica, san=
s-serif"><br></font></div><div style=3D"color:rgb(0,0,0)"><font face=3D"ari=
al, helvetica, sans-serif">to stop a seemingly forever-going repair but see=
ing really odd behavior with C* 2.0.9. Here is what I did:</font></div><div=
><font face=3D"arial, helvetica, sans-serif" style=3D"color:rgb(0,0,0)">1. =
First, I run &#39;nodetool tpstats&#39; on all nodes in the cluster and see=
ing only one node have 1 active pending=C2=A0</font><font color=3D"#000000"=
 face=3D"arial, helvetica, sans-serif">AntiEntropySessions. All other nodes=
 do not have any pending or active=C2=A0AntiEntropySessions.</font></div><d=
iv><font color=3D"#000000" face=3D"arial, helvetica, sans-serif">2. Then I =
grep &#39;Repair&#39; on all logs on all nodes and seeing absolutely no rep=
air related activity in these logs for the past day.</font></div><div><font=
 face=3D"arial, helvetica, sans-serif"><font color=3D"#000000">3. Then on t=
he node that has active=C2=A0AntiEntropySessions, I did steps &#39;a&#39; a=
nd &#39;b&#39; above. Now all the sudden I start seeing repair activities, =
on nodes that did not have pending=C2=A0</font><span style=3D"color:rgb(0,0=
,0)">AntiEntropySessions, I am seeing the following in their logs:</span></=
font></div><div><font face=3D"arial, helvetica, sans-serif"><font color=3D"=
#000000">INFO [NonPeriodicTasks:1] 2016-09-29 17:12:53,469 StreamingRepairT=
ask.java (line 87) [repair #e80e17d0-8667-11e6-a801-e172d7a67134] streaming=
 task succeed, returning response to /<a href=3D"http://10.253.2.166">10.25=
3.2.166</a></font><br></font></div><div><font face=3D"arial, helvetica, san=
s-serif"><font color=3D"#000000">On node=C2=A0</font><span style=3D"color:r=
gb(0,0,0)">10.253.2.166 which has active pending=C2=A0</span><span style=3D=
"color:rgb(0,0,0)">AntiEntropySessions, I am seeing the following in the lo=
g:</span></font></div><div><font face=3D"arial, helvetica, sans-serif"><fon=
t color=3D"#000000">INFO [AntiEntropySessions:136] 2016-09-29 17:03:02,405 =
RepairSession.java (line 282) [repair #812dafe0-8666-11e6-a801-e172d7a67134=
] session completed successfully</font><br></font></div><div><font color=3D=
"#000000" face=3D"arial, helvetica, sans-serif"><br></font></div><div><font=
 face=3D"arial, helvetica, sans-serif"><font color=3D"#000000">So it seems =
to me that by doing=C2=A0</font><span style=3D"color:rgb(0,0,0)">forceTermi=
nateAllRepairSession</span><wbr style=3D"color:rgb(0,0,0)"><span style=3D"c=
olor:rgb(0,0,0)">s, it actually &#39;wakes up&#39; the dormant repair so it=
 goes again. So far, the only way I can get working to stop a repair is to =
restart C* node where the repair command is initiated.</span></font></div><=
div><font face=3D"arial, helvetica, sans-serif"><span style=3D"color:rgb(0,=
0,0)"><br></span></font></div><div><font color=3D"#000000" face=3D"arial, h=
elvetica, sans-serif">Thanks.</font></div><div><font color=3D"#000000" face=
=3D"arial, helvetica, sans-serif"><br></font></div><div><font color=3D"#000=
000" face=3D"arial, helvetica, sans-serif">George.</font></div></div><div c=
lass=3D"gmail_extra"><br><div class=3D"gmail_quote">On Fri, Sep 23, 2016 at=
 6:20 AM, Romain Hardouin <span dir=3D"ltr">&lt;<a href=3D"mailto:romainh_m=
l@yahoo.fr" target=3D"_blank">romainh_ml@yahoo.fr</a>&gt;</span> wrote:<br>=
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div><div style=3D"color:#000;background-col=
or:#fff;font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Gra=
nde,sans-serif;font-size:16px"><div dir=3D"ltr"><font size=3D"2">OK. If you=
 still have issues after setting streaming_socket_timeout_in_ms !=3D 0, con=
sider increasing request_timeout_in_ms to a high value, say 1 or 2 minutes.=
 See comments in <a href=3D"https://issues.apache.org/jira/browse/CASSANDRA=
-7904" target=3D"_blank">https://issues.apache.org/<wbr>jira/browse/CASSAND=
RA-7904</a></font></div><div dir=3D"ltr"><font size=3D"2">Regarding 2.1, be=
 sure to test incremental repair on your data before to run it in productio=
n ;-)</font></div><span class=3D"HOEnZb"><font color=3D"#888888"><div dir=
=3D"ltr"><font size=3D"2"><br></font></div><div dir=3D"ltr"><font size=3D"2=
">Romain</font></div></font></span></div></div></blockquote></div><br></div=
>

--001a1145f7ce46e24f053da8ae36--