hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: How to replace MetaUtils.ScannerListener?
Date Fri, 22 Mar 2013 15:19:15 GMT
Let me offer a counter argument.  The offline splitting code is still
present even though online splitting isn't the problem it used to be. We
actually added an extenal wal replayer even though we have wal replay in
our normal recovery path.  Copy table still exists even though snapshots
and snapshot export exists.  Would we consider removing these?

IMO, until online merge has been proven and hardened (gone through few
releases and folk using) I don't feel comfortable removing it.

Jon.

On Thu, Mar 21, 2013 at 7:07 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> Hi Enis,
>
> I totaly agree. But even if online merge are available, maybe offline
> merges can still usefull in case the cluster is down for maintenance,
> or because there is any issue to start it, or anything else? We have
> it, so we should maybe try to keep it?
>
> JM
>
> 2013/3/21 Enis Söztutar <enis.soz@gmail.com>:
> > Thanks J-M.
> >
> > What I am trying to understand is that whether we should cut the cord for
> > offline merge once online is working. If you think about it, there should
> > not be a need to merge offline tables.
> >
> > Enis
> >
> >
> > On Thu, Mar 21, 2013 at 3:16 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> Offline merge is already there and working fine.
> >>
> >> The usecase here was to retreive all the regions for a given table to
> >> merge them 2 by 2, offline.
> >>
> >> It's working fine, but since the Meta rework it's not working anymore
> >> and I'm trying to rebase the patch.
> >>
> >> Like J-D is saying, yes, it's used only in the offline merge... And
> >> since the online merge is coming, I think it's cleaner to keep the
> >> code in the offline merge since it will disapear soon, but in the
> >> meantime, at least, we will have the offline one.
> >>
> >> JM
> >>
> >> 2013/3/21 Enis Söztutar <enis.soz@gmail.com>:
> >> > What is the use case behind offline merge? Is it because we cannot do
> >> > online merge yet? If we can get HBASE-7403 in, is there still need to
> >> > support offline merge?
> >> >
> >> > Enis
> >> >
> >> >
> >> > On Thu, Mar 21, 2013 at 2:56 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> >> >wrote:
> >> >
> >> >> As far as I can tell, only the merge code uses MetaUtils to do
> offline
> >> >> work. If this is the code you are in then pull it back into MetaUtils
> >> >> I think.
> >> >>
> >> >> J-D
> >> >>
> >> >> On Thu, Mar 21, 2013 at 2:44 PM, Jean-Marc Spaggiari
> >> >> <jean-marc@spaggiari.org> wrote:
> >> >> > Vector is because of a very old bad habit ;) I will change that
to
> >> >> ArrayList.
> >> >> >
> >> >> > So far I have inlined the scanMetaRegion feature into the Merge,
> but
> >> >> > maybe it should be cleaner to put it back in?
> >> >> >
> >> >> > Anyway, I will keep the inlined one until everything is cleaned.
> >> >> >
> >> >> > 2013/3/21 Enis Söztutar <enis.soz@gmail.com>:
> >> >> >> The problem around current META scanning is that there is
more
> than
> >> one
> >> >> way
> >> >> >> to do these, and the META layout is exposed. We should refrain
> from
> >> >> >> exposing the META details.
> >> >> >> AFAIK, these do the same thing:
> >> >> >>  MetaReader.Visitor
> >> >> >>  MetaScanner.MetaScannerVisitor, and
> >> >> >>  MetaUtils.ScannerListener
> >> >> >>
> >> >> >> More concerning is that the code for managing META is spread
over
> >> >> >> MetaEditor, MetaReader, MetaScanner, MetaUtils, HRegionInfo
(and
> >> maybe
> >> >> >> more). There are a couple of issues to rework these interfaces,
> but I
> >> >> did
> >> >> >> not get the chance to work on those.
> >> >> >>
> >> >> >>
> >> >> >> Enis
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Mar 21, 2013 at 2:18 PM, Jean-Daniel Cryans <
> >> >> jdcryans@apache.org>wrote:
> >> >> >>
> >> >> >>> Mmmm I may have been trigger happy. You could pull back
the
> >> >> >>> ScannerListener and scanMetaRegion.
> >> >> >>>
> >> >> >>> BTW, why are you using a Vector instead of ArrayList?
> >> >> >>>
> >> >> >>> J-D
> >> >> >>>
> >> >> >>> On Thu, Mar 21, 2013 at 2:05 PM, Jean-Marc Spaggiari
> >> >> >>> <jean-marc@spaggiari.org> wrote:
> >> >> >>> > Hi,
> >> >> >>> >
> >> >> >>> > In trunk, since HBASE-3171 (Drop ROOT and instead
store META
> >> >> >>> > location(s) directly in ZooKeeper ) there is no more
> >> >> >>> > MetaUtils.ScannerListener.
> >> >> >>> >
> >> >> >>> > In the merge, I used it to retreive all the regions
belonging
> to a
> >> >> >>> > specific table, from the META. kind of scan.
> >> >> >>> >
> >> >> >>> >           // Retrieve the list of regions for this
table.
> >> >> >>> >           final List<HRegionInfo> regions =
new
> >> >> Vector<HRegionInfo>();
> >> >> >>> >
> utils.scanMetaRegion(HRegionInfo.FIRST_META_REGIONINFO,
> >> new
> >> >> >>> > MetaUtils.ScannerListener() {
> >> >> >>> >             public boolean processRow(HRegionInfo
info) {
> >> >> >>> >               if ((info != null) &&
> >> >> >>> > (Bytes.compareTo(info.getTableName(), tableName)
== 0)) {
> >> >> >>> >                 regions.add(info);
> >> >> >>> >               }
> >> >> >>> >               return true;
> >> >> >>> >             }
> >> >> >>> >           });
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > Is there a recommanded way to replace that? The Merge
is
> running
> >> >> >>> > offline, so I can't do a scan.
> >> >> >>> >
> >> >> >>> > Thanks,
> >> >> >>> >
> >> >> >>> > JM
> >> >> >>>
> >> >>
> >>
>



-- 
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message