hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: hbase cluster high loads
Date Wed, 26 Sep 2012 19:35:26 GMT
On Wed, Sep 26, 2012 at 1:13 AM, Yusup Ashrap <aphawk@gmail.com> wrote:
> hi Stack, thanks for reply.
> *hbase version* is 0.90.2.

This is an extremely old version.

 we use ganglia to monitor our cluster.write/read
> is normal/equally distributed all day long. 1k write , 4k read.
> it's kinda impossible to upgrade at  current moment , and we dont have
> extra machines to cope with this situation.(migrate=>upgrade)

More machines at least would get you breathing room.

> those metrics are region server metrics.high load time is  about 10:30am  ~
> 11:00 am (I'm from China) today.
> all servers in the cluster are dedicated for hbase storage and we dont have
> any other jobs or programs running on those servers.
>
> I found that I have one major compaction going on that period,is it the
> main reason of high load situation?

Probably.  And compactions in 0.90.x are not too smart compared to
newer versions.

> and why does  this " Block cache LRU eviction " happen so frequently?
> #org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction
> started; Attempting to free 634.89 MB of total=5.27 GB
>

That's 'normal'.  Its a cache.  Its evicting least recently used
blocks to make room for more recently read blocks.

> *10:30's major compaction log*:
> 2012-09-26 10:31:36,581 DEBUG org.apache.hadoop.hbase.regionserver.Store:
> Major compaction triggered on store data; time since last major compaction
> 81842121ms
> 2012-09-26 10:31:36,581 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
> requested for
>

Look at your ganglia.  Do you see your iowait go up and your read
latency too at this time when the major compaction runs?

St.Ack

Mime
View raw message