hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timo Schaepe <t...@timoschaepe.de>
Subject Re: Problems with hbase.hregion.max.filesize
Date Wed, 18 Dec 2013 05:37:46 GMT
Hey Ted Yu,

I had digging the name node log and so far I've found nothing special. No Exception, FATAL
or ERROR message nor anything other peculiarities.
Only I see a lot of messages like this:

2013-12-12 13:53:22,541 INFO org.apache.hadoop.hdfs.StateChange: Removing lease on  /hbase/Sessions_1091/d04cadb1b2252dafc476c138e9651ca7/.splits/9717de41277e207c24359a18dae72cd3/l/58ab2c11ca9b4b4994ce54bac0bb4c68.d04cadb1b2252dafc476c138e9651ca7
from client DFSClient_hb_rs_baur-hbase7.baur.boreus.de,60020,1386712527761_1295065721_26
2013-12-12 13:53:22,541 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /hbase/Sessions_1091/d04cadb1b2252dafc476c138e9651ca7/.splits/9717de41277e207c24359a18dae72cd3/l/58ab2c11ca9b4b4994ce54bac0bb4c68.d04cadb1b2252dafc476c138e9651ca7
is closed by DFSClient_hb_rs_baur-hbase7.baur.boreus.de,60020,1386712527761_1295065721_26

But maybe that is normal. If you wanna have a look, you can find the log snippet at https://www.dropbox.com/s/8sls714knn4yqp3/hadoop-hadoop-namenode-baur-hbase1.log.2013-12-12.snip

Thanks,

	Timo



Am 14.12.2013 um 09:12 schrieb Ted Yu <yuzhihong@gmail.com>:

> Timo:
> Other than two occurrences of 'Took too long to split the files'
> @ 13:54:20,194 and 13:55:10,533, I don't find much clue from the posted log.
> 
> If you have time, mind checking namenode log for 1 minute interval leading
> up to 13:54:20,194 and 13:55:10,533, respectively ?
> 
> Thanks
> 
> 
> On Sat, Dec 14, 2013 at 5:21 AM, Timo Schaepe <timo@timoschaepe.de> wrote:
> 
>> Hey,
>> 
>> @JM: Thanks for the hint with hbase.regionserver.fileSplitTimeout. At the
>> moment (the import is actually working) and after I splittet the specific
>> regions manually, we do not have growing regions anymore.
>> 
>> hbase hbck says, all things are going fine.
>> 0 inconsistencies detected.
>> Status: OK
>> 
>> @Ted Yu: Sure, have a look here: http://pastebin.com/2ANFVZEU
>> The relevant tablename ist data_1091.
>> 
>> Thanks for your time.
>> 
>>        Timo
>> 
>> Am 13.12.2013 um 20:18 schrieb Ted Yu <yuzhihong@gmail.com>:
>> 
>>> Timo:
>>> Can you pastebin regionserver log around 2013-12-12 13:54:20 so that we
>> can
>>> see what happened ?
>>> 
>>> Thanks
>>> 
>>> 
>>> On Fri, Dec 13, 2013 at 11:02 AM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>> 
>>>> Try to increase hbase.regionserver.fileSplitTimeout but put it back to
>> its
>>>> default value after.
>>>> 
>>>> Default value is 30 seconds. I think it's not normal for a split to take
>>>> more than that.
>>>> 
>>>> What is your hardware configuration?
>>>> 
>>>> Have you run hbck to see if everything is correct?
>>>> 
>>>> JM
>>>> 
>>>> 
>>>> 2013/12/13 Timo Schaepe <timo@timoschaepe.de>
>>>> 
>>>>> Hello again,
>>>>> 
>>>>> digging in the logs of the specific regionserver shows me that:
>>>>> 
>>>>> 2013-12-12 13:54:20,194 INFO
>>>>> org.apache.hadoop.hbase.regionserver.SplitRequest: Running
>>>> rollback/cleanup
>>>>> of failed split of
>>>>> 
>>>> 
>> data,OR\x83\xCF\x02\x82\xAE\xF3U,1386851456415.d04cadb1b2252dafc476c138e9651ca7.;
>>>>> Took too long to split the files and create the references, aborting
>>>> split
>>>>> 
>>>>> This message appears two time, so it seems, that HBase tried to split
>> the
>>>>> region but it failed. I don't know why. How is the behaviour of HBase,
>>>> if a
>>>>> region split fails? Are there more tries to split this region again?
I
>>>>> didn't find any new tries in the log. Now I split the big regions
>>>> manually
>>>>> and this works. And also it seems, that HBase split the new regions
>> again
>>>>> to crunch they down to the given limit.
>>>>> 
>>>>> But also it is a mystery for me, why the split size in Hannibal shows
>> me
>>>>> 10 GB and in base-site.xml I put 2 GB…
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>>       Timo
>>>>> 
>>>>> 
>>>>> Am 13.12.2013 um 10:22 schrieb Timo Schaepe <timo@timoschaepe.de>:
>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> during the loading of data in our cluster I noticed some strange
>>>>> behavior of some regions, that I don't understand.
>>>>>> 
>>>>>> Scenario:
>>>>>> We convert data from a mysql database to HBase. The data is inserted
>>>>> with a put to the specific HBase table. The row key is a timestamp. I
>>>> know
>>>>> the problem with timestamp keys, but in our requirement it works quiet
>>>>> well. The problem is now, that there are some regions, which are
>> growing
>>>>> and growing.
>>>>>> 
>>>>>> For example the table on the picture [1]. First, all data was
>>>>> distributed over regions and node. And now, the data is written into
>> only
>>>>> one region, which is growing and I can see no splitting at all.
>> Actually
>>>>> the size of the big region is nearly 60 GB.
>>>>>> 
>>>>>> HBase version is 0.94.11. I cannot understand, why the splitting
is
>> not
>>>>> happening. In hbase-site.xml I limit the hbase.hregion.max.filesize to
>> 2
>>>> GB
>>>>> and HBase accepted this value.
>>>>>> 
>>>>>> <property>
>>>>>>     <!--Loaded from hbase-site.xml-->
>>>>>>     <name>hbase.hregion.max.filesize</name>
>>>>>>     <value>2147483648</value>
>>>>>> </property>
>>>>>> 
>>>>>> First mystery: Hannibal shows me the split size is 10 GB (see
>>>>> screenshot).
>>>>>> Second mystery: HBase is not splitting some regions neither at 2
GB
>> nor
>>>>> 10 GB.
>>>>>> 
>>>>>> Any ideas? Could be the timestamp rowkey cause this problem?
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>>     Timo
>>>>>> 
>>>>>> [1] https://www.dropbox.com/s/lm286xkcpglnj1t/big_region.png
>>>>> 
>>>>> 
>>>> 
>> 
>> 


Mime
View raw message