Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <CACCLA54gLnD=KLCSPM6KRg5PQ7yHjPmK7wQYZF+Cqz=pVN0A_A@mail.gmail.com>
References: <CACCLA57N-Ljjf+d-vtEHyqs+ePCBYVbRotZc6pWYZQkkLs6uOw@mail.gmail.com>
 <CACYuknYJJtbnDOEToAkPhHp4X95Bpwt8tZPaeMBgV3ge8DS1uQ@mail.gmail.com>
 <CACCLA54N_N-gXiqiwYBZSdsXkfRUMP=xMAkAG7zKhM6NCXZQDQ@mail.gmail.com>
 <CACYuknZCpo9zCbwSH8-QKoqeaKXW26G+LsxdTVfhhWa_Fu4U+A@mail.gmail.com> <CACCLA54gLnD=KLCSPM6KRg5PQ7yHjPmK7wQYZF+Cqz=pVN0A_A@mail.gmail.com>
From: Surbhi Gupta <surbhi.gupta01@gmail.com>
Date: Mon, 8 Aug 2016 13:52:53 -0700
Message-ID: <CACYuknbunewFu_neRM3X8CrKzC=fryROWWU0k2e5EeR8gWURJg@mail.gmail.com>
Subject: Re: compactions stuck and restart doesn't kill it
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a113de2b69db76105399599d4
archived-at: Mon, 08 Aug 2016 20:53:00 -0000

--001a113de2b69db76105399599d4
Content-Type: text/plain; charset=UTF-8

I have faced similar kind of issue while we were using LCS ...
We ran the scrub and it was helpful to clear the corruption and compaction
caught up well after that ...


On 8 August 2016 at 13:23, John Wong <gokoproject@gmail.com> wrote:

> Hi.
>
> I don't see any error logs indicate corrupted sstable error. Is it safe to
> run this?
>
> Thanks.
>
> On Mon, Aug 8, 2016 at 1:18 PM, Surbhi Gupta <surbhi.gupta01@gmail.com>
> wrote:
>
>> Can you see if any of the sstable is corrupt ?
>> I have seen with my past experience  , if any of the sstable which is
>> part of the compaction is corrupt then compaction kind of hangs.
>> Try to run nodetool scrub ...
>>
>> On 8 August 2016 at 09:34, John Wong <gokoproject@gmail.com> wrote:
>>
>>> On Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta <surbhi.gupta01@gmail.com>
>>> wrote:
>>>
>>>> Once you restart a node compaction will start automatically, if u dont
>>>> want to do so then do
>>>> nodetool disableautocompaction as soon as node is started .
>>>>
>>>>
>>> Thanks. I certainly can give that a try for the specific column family,
>>> but even so this would be a temporary solution, as the root issue is the
>>> compaction won't finish. I see nothing in the log. FYI, gathering more info
>>> from the developer, this CF stores large rows.
>>>
>>> If anyone ever encountered similar issue, please let me know.
>>>
>>> John
>>>
>>
>>
>

--001a113de2b69db76105399599d4
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I have faced similar kind of issue while we were using LCS=
 ...<div>We ran the scrub and it was helpful to clear the corruption and co=
mpaction caught up well after that ...</div><div><br></div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">On 8 August 2016 at 13:23, =
John Wong <span dir=3D"ltr">&lt;<a href=3D"mailto:gokoproject@gmail.com" ta=
rget=3D"_blank">gokoproject@gmail.com</a>&gt;</span> wrote:<br><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid=
;padding-left:1ex"><div dir=3D"ltr"><div><div>Hi.<br><br></div>I don&#39;t =
see any error logs indicate corrupted sstable error. Is it safe to run this=
?<br><br></div>Thanks.<br></div><div class=3D"HOEnZb"><div class=3D"h5"><di=
v class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Aug 8, 2016 =
at 1:18 PM, Surbhi Gupta <span dir=3D"ltr">&lt;<a href=3D"mailto:surbhi.gup=
ta01@gmail.com" target=3D"_blank">surbhi.gupta01@gmail.com</a>&gt;</span> w=
rote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;borde=
r-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Can you see if any=
 of the sstable is corrupt ?=C2=A0<div>I have seen with my past experience =
=C2=A0, if any of the sstable which is part of the compaction is corrupt th=
en compaction kind of hangs.</div><div>Try to run=C2=A0nodetool scrub ...</=
div></div><div><div><div class=3D"gmail_extra"><br><div class=3D"gmail_quot=
e">On 8 August 2016 at 09:34, John Wong <span dir=3D"ltr">&lt;<a href=3D"ma=
ilto:gokoproject@gmail.com" target=3D"_blank">gokoproject@gmail.com</a>&gt;=
</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .=
8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><span>On =
Mon, Aug 8, 2016 at 11:16 AM, Surbhi Gupta <span dir=3D"ltr">&lt;<a href=3D=
"mailto:surbhi.gupta01@gmail.com" target=3D"_blank">surbhi.gupta01@gmail.co=
m</a>&gt;</span> wrote:<br></span><div class=3D"gmail_extra"><div class=3D"=
gmail_quote"><span><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Once you=
 restart a node compaction will start automatically, if u dont want to do s=
o then do=C2=A0<div>nodetool disableautocompaction as soon as node is start=
ed .</div></div><div><div><div class=3D"gmail_extra"><br></div></div></div>=
</blockquote><div><br></div></span><div>Thanks. I certainly can give that a=
 try for the specific column family, but even so this would be a temporary =
solution, as the root issue is the compaction won&#39;t finish. I see nothi=
ng in the log. FYI, gathering more info from the developer, this CF stores =
large rows.<br><br></div><div>If anyone ever encountered similar issue, ple=
ase let me know.<span><font color=3D"#888888"><br></font></span></div><span=
><font color=3D"#888888"><div><br></div><div>John</div></font></span></div>=
</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a113de2b69db76105399599d4--