hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: splits and merge
Date Thu, 16 Jul 2015 13:50:32 GMT
Ivan,

Is it possible that you poll for number of regions (e.g. in a loop) after
invoking split or merge to confirm that the action has been performed? I
know it is a crude way but maybe something can be done in these lines. Or
are you already doing this when you said 'look for Region Info'?

Regards,
Shahab

On Thu, Jul 16, 2015 at 9:46 AM, Ivan Brondino <ibrondino@fi.upm.es> wrote:

> Hi,
>
> I can confirm that it was what Jean Marc was saying. Thanks JM!
>
> Regarding question #1, any help? I am rephrasing it in case was not clear.
>
> 1) Since both split and merge are async invocations on the client side, we
> have a naîve
> workaround to make it synchronous that is to look at the RS Info Server via
> an HttpClient checking
> for pending tasks to be finished. Once all tasks are finished in all RS we
> issue the next split/merge for the same table. Is there a better way to
> check if major compact has finished?
>
> Thanks again,
>
> Iván
>
>
> 2015-07-15 17:49 GMT+02:00 Ivan Brondino <ibrondino@fi.upm.es>:
>
> > Hi Ted, Jean Marc,
> >
> > Thank you guys for your answer.
> >
> > I think Jean Marc spotted the issue, although we do slightly different.
> >
> > Let's say we have regions A and B. We split both A  into A1 and A2 and B
> > into B1 and B2.
> > Then we merge A2 and B1. Note that A and B are adjacent and so A1 with
> A2,
> > A2 with B1 and B1 with B2.
> >
> > We were not waiting for the major compact after the splits. Thus it could
> > be the reason.
> >
> > We will give it a try and come back to you.
> >
> > Thanks,
> >
> > Ivan
> >
> >
> >
> >
> >
> > 2015-07-15 16:54 GMT+02:00 Jean-Marc Spaggiari <jean-marc@spaggiari.org
> >:
> >
> >> Hi Ivan,
> >>
> >>
> >> For #1, let's consider regions A, B, C and D all in server S.
> >>
> >> If you merge A and B togher and C and D togher, still in S, you will
> have
> >> A' and C' regions only on S.
> >>
> >> Now, if you try to merge again A' and C' into a new bigger region, this
> >> will fail silently until A' and C' are major compacted.
> >>
> >> Indeed, before major compaction, A' and C' will both still contain
> >> references to A+B and C+D files/regions. So to be able to merge A' and
> C'
> >> you need this to be cleaned and so you need to major compact them.
> >>
> >> HTH,
> >>
> >> JM
> >>
> >> 2015-07-15 10:43 GMT-04:00 Ted Yu <yuzhihong@gmail.com>:
> >>
> >> > bq. that some times merge just does not work.
> >> >
> >> > Can you identify under what scenario the merge doesn't work (through
> >> closer
> >> > inspection of the region server log - assuming you have DEBUG logging
> >> > turned on) ?
> >> >
> >> > bq. Are there minimum requirements for two regions to be merged?
> >> >
> >> > If the two adjacent regions to be merged are on different servers,
> HBase
> >> > would move one of them to the other region's server.
> >> >
> >> > bq. If those requirements are not met, does the merge process exit
> >> > silently?
> >> >
> >> > Nothing I am aware of.
> >> >
> >> > Cheers
> >> >
> >> > On Wed, Jul 15, 2015 at 3:08 AM, Ivan Brondino <ibrondino@fi.upm.es>
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > We are using Hbase 0.98.6 and Hadoop 2.5.0 - cdh 5.3.5.
> >> > >
> >> > > We have a couple of questions regarding splits and merge:
> >> > >
> >> > > 1) Since both split and merge are async invocations on the client
> >> side,
> >> > we
> >> > > have a naîve
> >> > > workaround to make it synchronous that is to look at the RS Info
> >> Server
> >> > via
> >> > > an HttpClient checking
> >> > > for pending tasks to be finished. Once all tasks are finished in all
> >> RS
> >> > we
> >> > > issue the next split/merge for the same table. Is there a better way
> >> to
> >> > do
> >> > > it?
> >> > >
> >> > > 2) Regarding merges, we've noticed that some times merge just does
> not
> >> > > work. We see on the master log that merge request is issued and it
> >> > forwards
> >> > > it to the corresponding RS but then on the RS nothing happens. We
> >> always
> >> > > merge two regions on the same region server.
> >> > > Are there minimum requirements for two regions to be merged? If
> those
> >> > > requirements are not met, does the merge process exit silently?
> >> > >
> >> > > Thanks in advance,
> >> > >
> >> > > Iván
> >> > >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message