accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ed Coleman" <d...@etcoleman.com>
Subject RE: Corrupt WAL
Date Tue, 21 Aug 2018 22:57:52 GMT
The has been work done in https://github.com/apache/accumulo/pull/574. I'm not certain of the
state of the code, but the description may provide you with things that you could look at
manually.


-----Original Message-----
From: tech.shan@gmail.com [mailto:tech.shan@gmail.com] 
Sent: Tuesday, August 21, 2018 5:45 PM
To: user@accumulo.apache.org
Subject: Re: Corrupt WAL

Was there any success with this workaround strategy?  I am also experiencing this issue.

On 2018/06/13 16:30:22, "Adam J. Shook" <adamjshook@gmail.com> wrote: 
> Sorry, I had the error backwards.  There is an OPEN for the WAL and 
> then immediately a COMPACTION_FINISH entry.  This would cause the error.
> 
> On Wed, Jun 13, 2018 at 11:34 AM, Adam J. Shook <adamjshook@gmail.com>
> wrote:
> 
> > Looking at the log I see that the last two entries are 
> > COMPACTION_START of one RFile immediately followed by a 
> > COMPACTION_START of a separate RFile which (I believe) would lead to 
> > the error.  Would this necessarily be an issue if the compactions are for separate
RFiles?
> >
> > This is a dev cluster and I don't necessarily care about it, but is 
> > there a (good) means to do WAL log surgery?  I imagine I can just 
> > chop off bytes until the log is parseable and missing the info about the compactions.
> >
> > On Tue, Jun 12, 2018 at 2:32 PM, Keith Turner <keith@deenlo.com> wrote:
> >
> >> On Tue, Jun 12, 2018 at 12:10 PM, Adam J. Shook 
> >> <adamjshook@gmail.com>
> >> wrote:
> >> > Yes, that is the error.  I'll inspect the logs and report back.
> >>
> >> Ok.  The LogReader command has a mechanism to filter which tablet 
> >> is displayed.  If the walog has  alot of data in it, may need to 
> >> use this.
> >>
> >> Also, be aware that only 5 mutations are shown for a "many mutations"
> >> objects in the walog.   The -m options changes this.  May want to see
> >> more when deciding if the info in the log is important.
> >>
> >>
> >> >
> >> > On Tue, Jun 12, 2018 at 10:14 AM, Keith Turner <keith@deenlo.com>
> >> wrote:
> >> >>
> >> >> Is the message you are seeing "COMPACTION_FINISH (without 
> >> >> preceding COMPACTION_START)" ?  That messages indicates that the 
> >> >> WALs are incomplete, probably as a result of the NN problems.  
> >> >> Could do the following :
> >> >>
> >> >> 1) Run the following command to see whats in the log.  Need to 
> >> >> see what is there for the root tablet.
> >> >>
> >> >>    accumulo org.apache.accumulo.tserver.logger.LogReader
> >> >>
> >> >> 2) Replace the log file with an empty file after seeing if there 
> >> >> is anything important in it.
> >> >>
> >> >> I think the list of WALs for the root tablet is stored in ZK at 
> >> >> /accumulo/<id>/walogs
> >> >>
> >> >> On Mon, Jun 11, 2018 at 5:26 PM, Adam J. Shook 
> >> >> <adamjshook@gmail.com>
> >> >> wrote:
> >> >> > Hey all,
> >> >> >
> >> >> > The root tablet on one of our dev systems isn't loading due to

> >> >> > an illegal state exception -- COMPACTION_FINISH preceding 
> >> >> > COMPACTION_START.
> >> What'd
> >> >> > be
> >> >> > the best way to mitigate this issue?  This was likely caused 
> >> >> > due to
> >> both
> >> >> > of
> >> >> > our NameNodes failing.
> >> >> >
> >> >> > Thank you,
> >> >> > --Adam
> >> >
> >> >
> >>
> >
> >
>


Mime
View raw message