hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geoff Hendrey" <ghend...@decarta.com>
Subject RE: extremely sluggish hbase
Date Tue, 20 Apr 2010 23:32:27 GMT
Yes, it shows BLOCKCACHE => 'false'

hbase(main):010:0> scan "-ROOT-"
ROW                          COLUMN+CELL
 .META.,,1                   column=info:regioninfo, timestamp=1271179314119, value=REGION
=> {NAME => '.META.
                             ,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,
TABLE => {{NAME => '.M
                             ETA.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
FAMILIES => [{NAME => 'h
                             istorian', VERSIONS => '2147483647', COMPRESSION => 'NONE',
TTL => '604800', BLOC
                             KSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}, {NAME => 'info', V
                             ERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647',
BLOCKSIZE => '8192',
                              IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
 .META.,,1                   column=info:server, timestamp=1271784127300, value=10.241.6.80:60020
 .META.,,1                   column=info:serverstartcode, timestamp=1271784127300, value=1271784122422


-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Tuesday, April 20, 2010 1:57 PM
To: hbase-user@hadoop.apache.org
Subject: Re: extremely sluggish hbase

Below it says blockcache is true for the 'info' family on .META.

What happens if you scan '-ROOT-', does it still stay info family has blockcache false?

Best way to get the script is to either update your hbase to 0.20.4 or just apply said patch.
 The script will then be in your bin directory.

St.Ack

On Tue, Apr 20, 2010 at 1:24 PM, Geoff Hendrey <ghendrey@decarta.com> wrote:
> Does look like the .META. BLOCKCACHE is false. What's the best way to 
> get a patch for https://issues.apache.org/jira/browse/HBASE-2451
>
> hbase(main):001:0> describe ".META."
> DESCRIPTION                                                  
          
> ENABLED
>  {NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384', 
> F true
>  AMILIES => [{NAME => 'historian', COMPRESSION => 'NONE', VERSIONS => '
>  2147483647', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 
> 'false
>  ', BLOCKCACHE => 'false'}, {NAME => 'info', COMPRESSION => 'NONE', 
> VER
>  SIONS => '10', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY =>
>  'false', BLOCKCACHE => 'true'}]}
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of 
> Stack
> Sent: Tuesday, April 20, 2010 12:45 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: extremely sluggish hbase
>
> If you scan '.META.' table is it slow also?  You could have a case of hbase-2451?  There
is a script in the patch to that issue.  Try it.
> See if that helps.
> St.Ack
>
> On Tue, Apr 20, 2010 at 12:02 PM, Geoff Hendrey <ghendrey@decarta.com> wrote:
>> Answers below, prefixed by "geoff:"
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of 
>> Stack
>> Sent: Tuesday, April 20, 2010 11:23 AM
>> To: hbase-user@hadoop.apache.org
>> Subject: Re: extremely sluggish hbase
>>
>> On Tue, Apr 20, 2010 at 10:29 AM, Geoff Hendrey <ghendrey@decarta.com> wrote:
>>> Hbase shell is taking 63 seconds to scan a table with {LIMIT=>1}!
>>
>> Is MR job running concurrently?
>> Geoff: no
>>
>> Whats happening on your servers?  High load?
>> Geoff: no, 99% idle on both servers
>>
>> I see
>>> this error occur frequently in the region server  logs. Any ideas on 
>>> what this might be>
>>>
>>> 2010-04-20 04:19:41,401 INFO org.apache.hadoop.ipc.HBaseServer: IPC 
>>> Server handler 2 on 60020, call next(-750587486574522252) from
>>> 10.241.6.80:51850: error:
>>> org.apache.hadoop.hbase.UnknownScannerException: Name:
>>> -750587486574522252
>>>
>>> I also see this in the regions server logs:
>>>
>>> 2010-04-20 04:21:44,559 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
>>> 5849633296569445699 lease expired
>>> 2010-04-20 04:21:44,560 INFO org.apache.hadoop.hdfs.DFSClient: Could 
>>> not obtain block blk_1799401938583830364_69702 from any node:
>>> java.io.IOException: No live nodes contain current block
>>>
>>
>>
>> So, this is usually because the client took long between 'next'
>> invocations on the scanner or the server is under such load its holding on to the
'next' call for so long that the next time 'next' is called, the scanner lease has expired.
>>
>>
>>> However "hadoop dfsadmin -report" doesn't show any HDFS issues. 
>>> Looks totally healthy. When I do "status" from HBase shell I get 
>>> "hbase(main):008:0> status
>>> 2 servers, 0 dead, 484.0000 average load" which also seems healthy 
>>> to me.
>>>
>>
>> Your servers are carrying 500 regions each.
>> Geoff: Is this high, moderate, or low for a typical installation?
>>
>>> Any suggestions?
>>>
>>
>> Look at top.  Look for loading.  Are you swapping?
>> Geoff: I will look into the swapping and see if I can get some numbers.
>>
>> Look in hbase logs.  Whats it say its doing?  Fat GC pauses?
>> Geoff: I monitor all the logs and I don't see any GC pauses. I am running 64 bit
java with 8GB of heap. I'll look into GC further and see if I can get some concrete data.
>>
>> St.Ack
>>
>

Mime
View raw message