hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Chan <tc...@prediction.io>
Subject Re: No serialized HRegionInfo
Date Wed, 03 Jun 2015 01:37:18 GMT
Thanks for the hint. I found from the log that regions assigned to a
particular regionserver (out of 3 machines) cause warning. I decommissioned
the regionserver in question and restarted it, fixed the inconsistencies
with -fixEmptyMetaCells and now the warning is gone.

Thanks,
Tom

On Sat, May 30, 2015 at 7:08 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> The pastebin of master log snippet doesn't provide enough clue.
>
> Can you trace, say, 1042331bd51614fabf01034113804665, back in master log to
> see whether it went through splits ?
> Please pastebin related portion of master log.
>
> Cheers
>
> On Fri, May 29, 2015 at 12:29 PM, Tom Chan <tchan@prediction.io> wrote:
>
> > On Thu, May 28, 2015 at 7:53 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > What release of hbase are you using ?
> > >
> > I'm using 0.98.9. I should have mentioned it before anything else.
> >
> > >
> > > If you trace 5fe111d3302cefb2e96d541a489ff026 in master log, do you
> find
> > > some clue ?
> > >
> > From the timestamp of the logs, all these warnings are given out when I
> do
> > "pio status" which tests if hbase is up by writing to it a table
> > "predictionio_eventdata:events_0" then deleting it.
> >
> > >
> > > Pastebin related log snippet if possible.
> > >
> >
> > http://pastebin.com/WDV01WQ7. There is only one line of "WARN No
> > HRegionInfo" instead of thousands of lines since I issued the "hbase hbck
> > -fixEmptyMetaCells" just now (but it keeps accumulating and that causes
> > problem).
> >
> > Also one thing I notice is that before fixing the meta table
> > (-fixEmptyMetaCells), in hbase shell, counting the non-existing table
> > 'predictionio_eventdata:events_0' takes much longer (a minute or two)
> than
> > some other non-existing tables. It spits out this line (or more) before
> > prompting that the table does not exist:
> >
> > ERROR: HRegionInfo was null or empty in hbase:meta,
> >
> >
> row=keyvalues={predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:seqnumDuringOpen/1432926325684/Put/vlen=8/mvcc=0,
> >
> >
> predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:server/1432926325684/Put/vlen=46/mvcc=0,
> >
> >
> predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:serverstartcode/1432926325684/Put/vlen=8/mvcc=0}
> >
> > Tom
> >
> >
> > >
> > > Thanks
> > >
> > > On Thu, May 28, 2015 at 6:48 PM, Tom Chan <tchan@prediction.io> wrote:
> > >
> > > > Hi,
> > > >
> > > > I kept running into this whenever I remove tables
> > > >
> > > > WARN catalog.MetaReader: No serialized HRegionInfo in
> > > >
> > > >
> > >
> >
> keyvalues={predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:seqnumDuringOpen/1430338711901/Put/vlen=8/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:server/1430338711901/Put/vlen=46/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:serverstartcode/1430338711901/Put/vlen=8/mvcc=0}
> > > >
> > > > create tables
> > > >
> > > > WARN client.HBaseAdmin: No serialized HRegionInfo in
> > > >
> > > >
> > >
> >
> keyvalues={predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:seqnumDuringOpen/1427845220983/Put/vlen=8/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:server/1427845220983/Put/vlen=46/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:serverstartcode/1427845220983/Put/vlen=8/mvcc=0}
> > > >
> > > >
> > > > When doing hbck, I found some 1800 inconsistencies of the form
> > > >
> > > > ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
> > > >
> > > >
> > > >
> > >
> >
> keyvalues={predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:seqnumDuringOpen/1431758974883/Put/vlen=8/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:server/1431758974883/Put/vlen=46/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:serverstartcode/1431758974883/Put/vlen=8/mvcc=0}
> > > >
> > > > while all the tables are okay.
> > > >
> > > > I ran "hbase hbck -fixEmptyMetaCells" according to the suggestion on
> > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201312.mbox/%3CCAPQV63VrpJ-_ffOo92m=RyaZa+Z3LrpPhiNMi5Szx5uar=u0rw@mail.gmail.com%3E
> > > > which
> > > > cleared the inconsistencies and those warning messages, but the "No
> > > > serialized HRegionInfo" and HBase inconsistencies continued to
> > accumulate
> > > > so some config is still not quite right. Any help is appreciated.
> > > >
> > > > Tom
> > > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message