hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Brondino <ibrond...@fi.upm.es>
Subject Re: splits and merge
Date Thu, 16 Jul 2015 13:46:49 GMT
Hi,

I can confirm that it was what Jean Marc was saying. Thanks JM!

Regarding question #1, any help? I am rephrasing it in case was not clear.

1) Since both split and merge are async invocations on the client side, we
have a naîve
workaround to make it synchronous that is to look at the RS Info Server via
an HttpClient checking
for pending tasks to be finished. Once all tasks are finished in all RS we
issue the next split/merge for the same table. Is there a better way to
check if major compact has finished?

Thanks again,

Iván


2015-07-15 17:49 GMT+02:00 Ivan Brondino <ibrondino@fi.upm.es>:

> Hi Ted, Jean Marc,
>
> Thank you guys for your answer.
>
> I think Jean Marc spotted the issue, although we do slightly different.
>
> Let's say we have regions A and B. We split both A  into A1 and A2 and B
> into B1 and B2.
> Then we merge A2 and B1. Note that A and B are adjacent and so A1 with A2,
> A2 with B1 and B1 with B2.
>
> We were not waiting for the major compact after the splits. Thus it could
> be the reason.
>
> We will give it a try and come back to you.
>
> Thanks,
>
> Ivan
>
>
>
>
>
> 2015-07-15 16:54 GMT+02:00 Jean-Marc Spaggiari <jean-marc@spaggiari.org>:
>
>> Hi Ivan,
>>
>>
>> For #1, let's consider regions A, B, C and D all in server S.
>>
>> If you merge A and B togher and C and D togher, still in S, you will have
>> A' and C' regions only on S.
>>
>> Now, if you try to merge again A' and C' into a new bigger region, this
>> will fail silently until A' and C' are major compacted.
>>
>> Indeed, before major compaction, A' and C' will both still contain
>> references to A+B and C+D files/regions. So to be able to merge A' and C'
>> you need this to be cleaned and so you need to major compact them.
>>
>> HTH,
>>
>> JM
>>
>> 2015-07-15 10:43 GMT-04:00 Ted Yu <yuzhihong@gmail.com>:
>>
>> > bq. that some times merge just does not work.
>> >
>> > Can you identify under what scenario the merge doesn't work (through
>> closer
>> > inspection of the region server log - assuming you have DEBUG logging
>> > turned on) ?
>> >
>> > bq. Are there minimum requirements for two regions to be merged?
>> >
>> > If the two adjacent regions to be merged are on different servers, HBase
>> > would move one of them to the other region's server.
>> >
>> > bq. If those requirements are not met, does the merge process exit
>> > silently?
>> >
>> > Nothing I am aware of.
>> >
>> > Cheers
>> >
>> > On Wed, Jul 15, 2015 at 3:08 AM, Ivan Brondino <ibrondino@fi.upm.es>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > We are using Hbase 0.98.6 and Hadoop 2.5.0 - cdh 5.3.5.
>> > >
>> > > We have a couple of questions regarding splits and merge:
>> > >
>> > > 1) Since both split and merge are async invocations on the client
>> side,
>> > we
>> > > have a naîve
>> > > workaround to make it synchronous that is to look at the RS Info
>> Server
>> > via
>> > > an HttpClient checking
>> > > for pending tasks to be finished. Once all tasks are finished in all
>> RS
>> > we
>> > > issue the next split/merge for the same table. Is there a better way
>> to
>> > do
>> > > it?
>> > >
>> > > 2) Regarding merges, we've noticed that some times merge just does not
>> > > work. We see on the master log that merge request is issued and it
>> > forwards
>> > > it to the corresponding RS but then on the RS nothing happens. We
>> always
>> > > merge two regions on the same region server.
>> > > Are there minimum requirements for two regions to be merged? If those
>> > > requirements are not met, does the merge process exit silently?
>> > >
>> > > Thanks in advance,
>> > >
>> > > Iván
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message