hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pere Kyle <p...@whisper.sh>
Subject Re: Hbase Unusable after auto split to 1024 regions
Date Thu, 06 Nov 2014 18:05:31 GMT
So I have set the heap to 12Gb and the memstore limit to upperLimit .5 lowerLimit .45. I am
not seeing any changes in behavior from the cluster so far, i have restarted 4/17 region servers.
Do I need to restart master for the memstore to take effect? Also how do I enable logging
to show why a region is being flushed? I don’t ever seen the region flushes in my logs.

Thanks,
Pere
On Nov 6, 2014, at 7:12 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> bq. to increase heap and increase the memstore limit?
> 
> Yes. That would be an action that bears fruit.
> Long term, you should merge the small regions.
> 
> Cheers
> 
> On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pere@whisper.sh> wrote:
> 
>> Watching closely a region server in action. It seems that the memstores
>> are being flushed at around  2MB on the regions. This would seem to
>> indicate that there is not enough heap for the memstore and I am hitting
>> the upper bound of limit (default). Would this be a fair assumption? Should
>> I look to increase heap and increase the memstore limit?
>> 
>> Thanks!
>> -Pere
>> 
>> On Nov 5, 2014, at 10:26 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> 
>>> You can use ConstantSizeRegionSplitPolicy.
>>> Split policy can be specified per table. See the following example
>>> in create.rb :
>>> 
>>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
>>> 'HexStringSplit'}
>>> 
>>> In 0.94.18, there isn't online merge. So you have to use other method to
>>> merge the small regions.
>>> 
>>> Cheers
>>> 
>>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pere@whisper.sh> wrote:
>>> 
>>>> Ted,
>>>> 
>>>> Thanks so much for that information. I now see why this split too often,
>>>> but what I am not sure of is how to fix this without blowing away the
>>>> cluster. Add more heap?
>>>> 
>>>> Another symptom I have noticed is that load on the Master instance hbase
>>>> daemon has been pretty high (load average 4.0, whereas it used to be
>> 1.0)
>>>> 
>>>> Thanks,
>>>> Pere
>>>> 
>>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>> 
>>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
>>>>> 
>>>>> You can read the javadoc of this class to see how it works.
>>>>> 
>>>>> Cheers
>>>>> 
>>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>>>> 
>>>>>> Can you provide a bit more information (such as HBase release) ?
>>>>>> 
>>>>>> If you pastebin one of the region servers' log, that would help us
>>>>>> determine the cause.
>>>>>> 
>>>>>> Cheers
>>>>>> 
>>>>>> 
>>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pere@whisper.sh>
wrote:
>>>>>> 
>>>>>>> Hello,
>>>>>>> 
>>>>>>> Recently our cluster which has been running fine for 2 weeks
split to
>>>>>>> 1024 regions at 1GB per region, after this split the cluster
is
>>>> unusable.
>>>>>>> Using the performance benchmark I was getting a little better
than
>> 100
>>>> w/s,
>>>>>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge
with
>>>> 8GB
>>>>>>> heap reserved for Hbase
>>>>>>> 
>>>>>>> Any Ideas? I am stumped:
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Pere
>>>>>>> 
>>>>>>> Here is the current
>>>>>>> hbase-site.xml
>>>>>>> <?xml version="1.0"?>
>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>>>> <configuration>
>>>>>>> <property>
>>>>>>>  <name>hbase.snapshot.enabled</name>
>>>>>>>  <value>true</value>
>>>>>>> </property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>>>>>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
>>>>>>> </value></property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>>>>>>> 
>>>>>>> 
>>>> 
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
>>>>>>> </configuration>
>>>>>>> 
>>>>>>> hbase-env.sh
>>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>>>>>> export HBASE_HEAPSIZE=8000
>>>>>>> 
>>>>>>> # Extra Java runtime options.
>>>>>>> # Below are what we set by default.  May only work with SUN JVM.
>>>>>>> # For more on why as well as other possible settings,
>>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>>>>>>> 
>>>>>>> hbase-env.sh
>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 


Mime
View raw message