hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: extremely sluggish hbase
Date Tue, 20 Apr 2010 18:22:40 GMT
On Tue, Apr 20, 2010 at 10:29 AM, Geoff Hendrey <ghendrey@decarta.com> wrote:
> Hbase shell is taking 63 seconds to scan a table with {LIMIT=>1}!

Is MR job running concurrently?

Whats happening on your servers?  High load?

I see
> this error occur frequently in the region server  logs. Any ideas on
> what this might be>
>
> 2010-04-20 04:19:41,401 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 2 on 60020, call next(-750587486574522252) from
> 10.241.6.80:51850: error:
> org.apache.hadoop.hbase.UnknownScannerException: Name:
> -750587486574522252
>
> I also see this in the regions server logs:
>
> 2010-04-20 04:21:44,559 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 5849633296569445699 lease expired
> 2010-04-20 04:21:44,560 INFO org.apache.hadoop.hdfs.DFSClient: Could not
> obtain block blk_1799401938583830364_69702 from any node:
> java.io.IOException: No live nodes contain current block
>


So, this is usually because the client took long between 'next'
invocations on the scanner or the server is under such load its
holding on to the 'next' call for so long that the next time 'next' is
called, the scanner lease has expired.


> However "hadoop dfsadmin -report" doesn't show any HDFS issues. Looks
> totally healthy. When I do "status" from HBase shell I get
> "hbase(main):008:0> status
> 2 servers, 0 dead, 484.0000 average load" which also seems healthy to
> me.
>

Your servers are carrying 500 regions each.

> Any suggestions?
>

Look at top.  Look for loading.  Are you swapping?  Look in hbase
logs.  Whats it say its doing?  Fat GC pauses?

St.Ack

Mime
View raw message