Mailing-List: contact user-help@kudu.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@kudu.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CADY20s7kf54Qoh+NpTw031hsxfJOy1XD4j83jXeD9X-yVe8sqg@mail.gmail.com>
References: 
 <CAGHRzmnWbbchGp07n7Nhm_UYdC+OJm=pDFC80e1pmYUWEOn5Jg@mail.gmail.com>
	<CADY20s7kf54Qoh+NpTw031hsxfJOy1XD4j83jXeD9X-yVe8sqg@mail.gmail.com>
Date: Fri, 19 Feb 2016 17:05:04 -0800
Message-ID: 
 <CAGHRzmmcTBpg+CDo5O75hzkqS8RRtC-CuFGAkR1rz+71AV1v6Q@mail.gmail.com>
Subject: Re: [KUDU Tablet]unrecoverable crash
From: Nick Wolf <nickwolf7@gmail.com>
To: user@kudu.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a113ed432977f50052c293097

--001a113ed432977f50052c293097
Content-Type: text/plain; charset=UTF-8

I've identified the tablet ID and tried to delete and start the server but
it seems like chain reaction. They keep coming one by one with different
tablet ids.
rm tablet-meta/1c2475126c7c4cc2b82f95bd6af5bdb4
rm wals/1c2475126c7c4cc2b82f95bd6af5bdb4
rm consensus-meta/1c2475126c7c4cc2b82f95bd6af5bdb4

A notable point here is none of the tablet ids that it shows bootstrapping
are not appearing in web interface. (http://host:8051/tables)

On Fri, Feb 19, 2016 at 12:49 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Nick,
>
> Are you able to determine the tablet ID that is failing to restart?
> The log line indicates that it's thread ID 6285. If you look farther
> up the log with 'grep " 6285 " kudu-tserver.INFO', you should see a
> log message indicating that that thread is starting to bootstrap a
> particular tablet.
>
> Is this a replicated table, or num_replicas=1? If it's replicated, we
> can probably recover by removing the corrupt replica and letting it
> grab a new copy from one of the other replicas. Otherwise, we'll have
> to do some more serious "surgery" which we can assist you with.
>
> Either way, see if you can figure out the bad tablet ID. Then, if it's
> possible to send a copy of the WAL directory for this tablet to me off
> list, I can try to do some post-mortem analysis to see what went
> wrong.
>
> Thanks
> -Todd
>
> On Fri, Feb 19, 2016 at 12:37 PM, Nick Wolf <nickwolf7@gmail.com> wrote:
> > KUDU Tablet crashed with following fatal error.
> >
> > F0219 12:15:11.389806  6285 mvcc.cc:542] Check failed: _s.ok() Bad
> status:
> > Illegal state: Timestamp: 5963266013874102274 is already committed.
> Current
> > Snapshot: MvccSnapshot[committed={T|T < 5963266013874118554 or (T in
> > {5963266013874118554})}]
> >
> > It throws the same fatal error and crashes immediately no matter how many
> > times i try to restart the service.
> >
> > Any ideas to get out of this situation? I don't want to lose the data.
> >
> >
> > --Nick
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--001a113ed432977f50052c293097
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I&#39;ve identified the tablet ID and tried to delete and =
start the server but it seems like chain reaction. They keep coming one by =
one with different tablet ids.<div>rm tablet-meta/1c2475126c7c4cc2b82f95bd6=
af5bdb4=C2=A0<div>rm wals/1c2475126c7c4cc2b82f95bd6af5bdb4</div><div>rm con=
sensus-meta/1c2475126c7c4cc2b82f95bd6af5bdb4</div></div><div><br></div><div=
>A notable point here is none of the tablet ids that it shows bootstrapping=
 are not appearing in web interface. (<a href=3D"http://host:8051/tables">h=
ttp://host:8051/tables</a>)</div></div><div class=3D"gmail_extra"><br><div =
class=3D"gmail_quote">On Fri, Feb 19, 2016 at 12:49 PM, Todd Lipcon <span d=
ir=3D"ltr">&lt;<a href=3D"mailto:todd@cloudera.com" target=3D"_blank">todd@=
cloudera.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" sty=
le=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Nic=
k,<br>
<br>
Are you able to determine the tablet ID that is failing to restart?<br>
The log line indicates that it&#39;s thread ID 6285. If you look farther<br=
>
up the log with &#39;grep &quot; 6285 &quot; kudu-tserver.INFO&#39;, you sh=
ould see a<br>
log message indicating that that thread is starting to bootstrap a<br>
particular tablet.<br>
<br>
Is this a replicated table, or num_replicas=3D1? If it&#39;s replicated, we=
<br>
can probably recover by removing the corrupt replica and letting it<br>
grab a new copy from one of the other replicas. Otherwise, we&#39;ll have<b=
r>
to do some more serious &quot;surgery&quot; which we can assist you with.<b=
r>
<br>
Either way, see if you can figure out the bad tablet ID. Then, if it&#39;s<=
br>
possible to send a copy of the WAL directory for this tablet to me off<br>
list, I can try to do some post-mortem analysis to see what went<br>
wrong.<br>
<br>
Thanks<br>
-Todd<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
On Fri, Feb 19, 2016 at 12:37 PM, Nick Wolf &lt;<a href=3D"mailto:nickwolf7=
@gmail.com">nickwolf7@gmail.com</a>&gt; wrote:<br>
&gt; KUDU Tablet crashed with following fatal error.<br>
&gt;<br>
&gt; F0219 12:15:11.389806=C2=A0 6285 mvcc.cc:542] Check failed: _s.ok() Ba=
d status:<br>
&gt; Illegal state: Timestamp: 5963266013874102274 is already committed. Cu=
rrent<br>
&gt; Snapshot: MvccSnapshot[committed=3D{T|T &lt; 5963266013874118554 or (T=
 in<br>
&gt; {5963266013874118554})}]<br>
&gt;<br>
&gt; It throws the same fatal error and crashes immediately no matter how m=
any<br>
&gt; times i try to restart the service.<br>
&gt;<br>
&gt; Any ideas to get out of this situation? I don&#39;t want to lose the d=
ata.<br>
&gt;<br>
&gt;<br>
&gt; --Nick<br>
&gt;<br>
<br>
<br>
<br>
</div></div><span class=3D"HOEnZb"><font color=3D"#888888">--<br>
Todd Lipcon<br>
Software Engineer, Cloudera<br>
</font></span></blockquote></div><br></div>

--001a113ed432977f50052c293097--