Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
From: Aiman Parvaiz <aiman@steelhouse.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: Need help with incremental repair
Thread-Topic: Need help with incremental repair
Thread-Index: AQHTT8Zf2WJG1Fzdhkyvj4dQZyk8hKL5bXyAgAHjygCAABOhog==
Date: Sun, 29 Oct 2017 22:12:49 +0000
Message-ID: <BN6PR08MB3362962381D37D253B1D66D2D0580@BN6PR08MB3362.namprd08.prod.outlook.com>
References: <BN6PR08MB3362BAD371880447F475F2AAD05B0@BN6PR08MB3362.namprd08.prod.outlook.com>
 <etPan.59f4aab1.74c1e1e4.74dc@apple.com>,<CAKaZCX7BX9f70u74xmcR8QGt4HtWXP0EEyna9AzRpJaORYdoow@mail.gmail.com>
In-Reply-To: <CAKaZCX7BX9f70u74xmcR8QGt4HtWXP0EEyna9AzRpJaORYdoow@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: steelhouse.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_BN6PR08MB3362962381D37D253B1D66D2D0580BN6PR08MB3362namp_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: cd0bc504-cabb-40bb-2a27-08d51f1a3175
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Oct 2017 22:12:49.9666
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 83691c01-760c-4b88-9cfb-a1b169bdb304
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR08MB3362
archived-at: Sun, 29 Oct 2017 22:13:04 -0000

--_000_BN6PR08MB3362962381D37D253B1D66D2D0580BN6PR08MB3362namp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Thanks Blake and Paulo for the response.

Yes, the idea is to go back to non incremental repairs. I am waiting for al=
l the "anticompaction after repair" activities to complete and in my unders=
tanding( thanks to Blake for the explanation ), I can run a full repair on =
that KS and then get back to my non incremental repair regiment.


I assume that I should mark the SSTs to un repaired first and then run a fu=
ll repair?

Also, although I am installing Cassandra from package dsc22 on my CentOS 7 =
I couldn't find sstable tools installed, need to figure that out too.

________________________________
From: Paulo Motta <pauloricardomg@gmail.com>
Sent: Sunday, October 29, 2017 1:56:38 PM
To: user@cassandra.apache.org
Subject: Re: Need help with incremental repair

> Assuming the situation is just "we accidentally ran incremental repair", =
you shouldn't have to do anything. It's not going to hurt anything

Once you run incremental repair, your data is permanently marked as
repaired, and is no longer compacted with new non-incrementally
repaired data. This can cause read fragmentation and prevent deleted
data from being purged. If you ever run incremental repair and want to
switch to non-incremental repair, you should manually mark your
repaired SSTables as not-repaired with the sstablerepairedset tool.

2017-10-29 3:05 GMT+11:00 Blake Eggleston <beggleston@apple.com>:
> Hey Aiman,
>
> Assuming the situation is just "we accidentally ran incremental repair", =
you
> shouldn't have to do anything. It's not going to hurt anything. Pre-4.0
> incremental repair has some issues that can cause a lot of extra streamin=
g,
> and inconsistencies in some edge cases, but as long as you're running ful=
l
> repairs before gc grace expires, everything should be ok.
>
> Thanks,
>
> Blake
>
>
> On October 28, 2017 at 1:28:42 AM, Aiman Parvaiz (aiman@steelhouse.com)
> wrote:
>
> Hi everyone,
>
> We seek your help in a issue we are facing in our 2.2.8 version.
>
> We have 24 nodes cluster spread over 3 DCs.
>
> Initially, when the cluster was in a single DC we were using The Last Pic=
kle
> reaper 0.5 to repair it with incremental repair set to false. We added 2
> more DCs. Now the problem is that accidentally on one of the newer DCs we
> ran nodetool repair <keyspace> without realizing that for 2.2 the default
> option is incremental.
>
> I am not seeing any errors in the logs till now but wanted to know what
> would be the best way to handle this situation. To make things a little m=
ore
> complicated, the node on which we triggered this repair is almost out of
> disk and we had to restart C* on it.
>
> I can see a bunch of "anticompaction after repair" under Opscenter Activi=
tes
> across various nodes in the 3 DCs.
>
>
> Any help, suggestion would be appreciated.
>
> Thanks
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


--_000_BN6PR08MB3362962381D37D253B1D66D2D0580BN6PR08MB3362namp_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; pad=
ding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content=3D"text/html; charset=3DUTF-8">
<style type=3D"text/css" style=3D"">
<!--
p
	{margin-top:0;
	margin-bottom:0}
-->
</style>
<div dir=3D"ltr">
<div id=3D"x_divtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt; col=
or:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>Thanks Blake and Paulo for the response.</p>
<p>Yes, the idea is to go back to non incremental repairs. I am waiting for=
 all the &quot;anticompaction after repair&quot; activities to complete and=
 in my understanding(&nbsp;thanks to Blake for the explanation&nbsp;), I ca=
n run a full repair on that KS and then get back to
 my non incremental repair regiment.</p>
<p><br>
</p>
<p><span style=3D"font-size:12pt">I assume that&nbsp;I should&nbsp;mark the=
 SSTs to un repaired first&nbsp;</span><span style=3D"font-size:12pt">and t=
hen run a full repair?&nbsp;</span></p>
<p><span style=3D"font-size:12pt">Also, although I am installing Cassandra =
from package dsc22 on my CentOS 7 I couldn't find sstable tools installed, =
need to figure that out too.</span><br>
</p>
</div>
<hr tabindex=3D"-1" style=3D"display:inline-block; width:98%">
<div id=3D"x_divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" =
color=3D"#000000" style=3D"font-size:11pt"><b>From:</b> Paulo Motta &lt;pau=
loricardomg@gmail.com&gt;<br>
<b>Sent:</b> Sunday, October 29, 2017 1:56:38 PM<br>
<b>To:</b> user@cassandra.apache.org<br>
<b>Subject:</b> Re: Need help with incremental repair</font>
<div>&nbsp;</div>
</div>
</div>
<font size=3D"2"><span style=3D"font-size:10pt;">
<div class=3D"PlainText">&gt; Assuming the situation is just &quot;we accid=
entally ran incremental repair&quot;, you shouldn't have to do anything. It=
's not going to hurt anything<br>
<br>
Once you run incremental repair, your data is permanently marked as<br>
repaired, and is no longer compacted with new non-incrementally<br>
repaired data. This can cause read fragmentation and prevent deleted<br>
data from being purged. If you ever run incremental repair and want to<br>
switch to non-incremental repair, you should manually mark your<br>
repaired SSTables as not-repaired with the sstablerepairedset tool.<br>
<br>
2017-10-29 3:05 GMT&#43;11:00 Blake Eggleston &lt;beggleston@apple.com&gt;:=
<br>
&gt; Hey Aiman,<br>
&gt;<br>
&gt; Assuming the situation is just &quot;we accidentally ran incremental r=
epair&quot;, you<br>
&gt; shouldn't have to do anything. It's not going to hurt anything. Pre-4.=
0<br>
&gt; incremental repair has some issues that can cause a lot of extra strea=
ming,<br>
&gt; and inconsistencies in some edge cases, but as long as you're running =
full<br>
&gt; repairs before gc grace expires, everything should be ok.<br>
&gt;<br>
&gt; Thanks,<br>
&gt;<br>
&gt; Blake<br>
&gt;<br>
&gt;<br>
&gt; On October 28, 2017 at 1:28:42 AM, Aiman Parvaiz (aiman@steelhouse.com=
)<br>
&gt; wrote:<br>
&gt;<br>
&gt; Hi everyone,<br>
&gt;<br>
&gt; We seek your help in a issue we are facing in our 2.2.8 version.<br>
&gt;<br>
&gt; We have 24 nodes cluster spread over 3 DCs.<br>
&gt;<br>
&gt; Initially, when the cluster was in a single DC we were using The Last =
Pickle<br>
&gt; reaper 0.5 to repair it with incremental repair set to false. We added=
 2<br>
&gt; more DCs. Now the problem is that accidentally on one of the newer DCs=
 we<br>
&gt; ran nodetool repair &lt;keyspace&gt; without realizing that for 2.2 th=
e default<br>
&gt; option is incremental.<br>
&gt;<br>
&gt; I am not seeing any errors in the logs till now but wanted to know wha=
t<br>
&gt; would be the best way to handle this situation. To make things a littl=
e more<br>
&gt; complicated, the node on which we triggered this repair is almost out =
of<br>
&gt; disk and we had to restart C* on it.<br>
&gt;<br>
&gt; I can see a bunch of &quot;anticompaction after repair&quot; under Ops=
center Activites<br>
&gt; across various nodes in the 3 DCs.<br>
&gt;<br>
&gt;<br>
&gt; Any help, suggestion would be appreciated.<br>
&gt;<br>
&gt; Thanks<br>
&gt;<br>
&gt;<br>
<br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org<br>
For additional commands, e-mail: user-help@cassandra.apache.org<br>
<br>
</div>
</span></font>
</body>
</html>

--_000_BN6PR08MB3362962381D37D253B1D66D2D0580BN6PR08MB3362namp_--