hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enis Söztutar <enis....@gmail.com>
Subject Re: offline merge Was: How to replace MetaUtils.ScannerListener?
Date Fri, 22 Mar 2013 00:03:28 GMT
I agree that for 0.94 we would still need offline merge, but if we can get
away with it, I would rather support only one tool.

Enis


On Thu, Mar 21, 2013 at 4:31 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Changing the subject to reflect current discussion.
>
> I think Jean-Marc's work on offline merge would most likely be useful in
> the context of 0.94.x code base.
>
> Cheers
>
> On Thu, Mar 21, 2013 at 4:17 PM, Enis Söztutar <enis.soz@gmail.com> wrote:
>
> > Thanks J-M.
> >
> > What I am trying to understand is that whether we should cut the cord for
> > offline merge once online is working. If you think about it, there should
> > not be a need to merge offline tables.
> >
> > Enis
> >
> >
> > On Thu, Mar 21, 2013 at 3:16 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > Offline merge is already there and working fine.
> > >
> > > The usecase here was to retreive all the regions for a given table to
> > > merge them 2 by 2, offline.
> > >
> > > It's working fine, but since the Meta rework it's not working anymore
> > > and I'm trying to rebase the patch.
> > >
> > > Like J-D is saying, yes, it's used only in the offline merge... And
> > > since the online merge is coming, I think it's cleaner to keep the
> > > code in the offline merge since it will disapear soon, but in the
> > > meantime, at least, we will have the offline one.
> > >
> > > JM
> > >
> > > 2013/3/21 Enis Söztutar <enis.soz@gmail.com>:
> > > > What is the use case behind offline merge? Is it because we cannot do
> > > > online merge yet? If we can get HBASE-7403 in, is there still need to
> > > > support offline merge?
> > > >
> > > > Enis
> > > >
> > > >
> > > > On Thu, Mar 21, 2013 at 2:56 PM, Jean-Daniel Cryans <
> > jdcryans@apache.org
> > > >wrote:
> > > >
> > > >> As far as I can tell, only the merge code uses MetaUtils to do
> offline
> > > >> work. If this is the code you are in then pull it back into
> MetaUtils
> > > >> I think.
> > > >>
> > > >> J-D
> > > >>
> > > >> On Thu, Mar 21, 2013 at 2:44 PM, Jean-Marc Spaggiari
> > > >> <jean-marc@spaggiari.org> wrote:
> > > >> > Vector is because of a very old bad habit ;) I will change that
to
> > > >> ArrayList.
> > > >> >
> > > >> > So far I have inlined the scanMetaRegion feature into the Merge,
> but
> > > >> > maybe it should be cleaner to put it back in?
> > > >> >
> > > >> > Anyway, I will keep the inlined one until everything is cleaned.
> > > >> >
> > > >> > 2013/3/21 Enis Söztutar <enis.soz@gmail.com>:
> > > >> >> The problem around current META scanning is that there is
more
> than
> > > one
> > > >> way
> > > >> >> to do these, and the META layout is exposed. We should refrain
> from
> > > >> >> exposing the META details.
> > > >> >> AFAIK, these do the same thing:
> > > >> >>  MetaReader.Visitor
> > > >> >>  MetaScanner.MetaScannerVisitor, and
> > > >> >>  MetaUtils.ScannerListener
> > > >> >>
> > > >> >> More concerning is that the code for managing META is spread
over
> > > >> >> MetaEditor, MetaReader, MetaScanner, MetaUtils, HRegionInfo
(and
> > > maybe
> > > >> >> more). There are a couple of issues to rework these interfaces,
> > but I
> > > >> did
> > > >> >> not get the chance to work on those.
> > > >> >>
> > > >> >>
> > > >> >> Enis
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Thu, Mar 21, 2013 at 2:18 PM, Jean-Daniel Cryans <
> > > >> jdcryans@apache.org>wrote:
> > > >> >>
> > > >> >>> Mmmm I may have been trigger happy. You could pull back
the
> > > >> >>> ScannerListener and scanMetaRegion.
> > > >> >>>
> > > >> >>> BTW, why are you using a Vector instead of ArrayList?
> > > >> >>>
> > > >> >>> J-D
> > > >> >>>
> > > >> >>> On Thu, Mar 21, 2013 at 2:05 PM, Jean-Marc Spaggiari
> > > >> >>> <jean-marc@spaggiari.org> wrote:
> > > >> >>> > Hi,
> > > >> >>> >
> > > >> >>> > In trunk, since HBASE-3171 (Drop ROOT and instead
store META
> > > >> >>> > location(s) directly in ZooKeeper ) there is no
more
> > > >> >>> > MetaUtils.ScannerListener.
> > > >> >>> >
> > > >> >>> > In the merge, I used it to retreive all the regions
belonging
> > to a
> > > >> >>> > specific table, from the META. kind of scan.
> > > >> >>> >
> > > >> >>> >           // Retrieve the list of regions for this
table.
> > > >> >>> >           final List<HRegionInfo> regions
= new
> > > >> Vector<HRegionInfo>();
> > > >> >>> >
> > utils.scanMetaRegion(HRegionInfo.FIRST_META_REGIONINFO,
> > > new
> > > >> >>> > MetaUtils.ScannerListener() {
> > > >> >>> >             public boolean processRow(HRegionInfo
info) {
> > > >> >>> >               if ((info != null) &&
> > > >> >>> > (Bytes.compareTo(info.getTableName(), tableName)
== 0)) {
> > > >> >>> >                 regions.add(info);
> > > >> >>> >               }
> > > >> >>> >               return true;
> > > >> >>> >             }
> > > >> >>> >           });
> > > >> >>> >
> > > >> >>> >
> > > >> >>> > Is there a recommanded way to replace that? The
Merge is
> running
> > > >> >>> > offline, so I can't do a scan.
> > > >> >>> >
> > > >> >>> > Thanks,
> > > >> >>> >
> > > >> >>> > JM
> > > >> >>>
> > > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message