hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akmal Abbasov <akmal.abba...@icloud.com>
Subject Re: High iowait in idle hbase cluster
Date Wed, 02 Sep 2015 16:39:15 GMT
Hi Ted,
sorry forget to mention

> release of hbase / hadoop you're using
hbase hbase-0.98.7-hadoop2, hadoop hadoop-2.5.1

> were region servers doing compaction ?
I’ve run major compactions manually earlier today, but it seems that they already completed,
looking at the compactionQueueSize.

> have you checked region server logs ?
The logs of datanode is full of this kind of messages
2015-09-02 16:37:06,950 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.10.8.55:50010, dest: /10.10.8.54:32959, bytes: 19673, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_1225374853_1,
offset: 0, srvID: ee7d0634-89a3-4ada-a8ad-7848217327be, blockid: BP-329084760-10.32.0.180-1387281790961:blk_1075277914_1540222,
duration: 7881815

p.s. we had to change the ip addresses of the cluster nodes, is it relevant?

Thanks.

> On 02 Sep 2015, at 18:20, Ted Yu <yuzhihong@gmail.com> wrote:
> 
> Please provide some more information:
> 
> release of hbase / hadoop you're using
> were region servers doing compaction ?
> have you checked region server logs ?
> 
> Thanks
> 
> On Wed, Sep 2, 2015 at 9:11 AM, Akmal Abbasov <akmal.abbasov@icloud.com <mailto:akmal.abbasov@icloud.com>>
wrote:
> Hi,
> I’m having strange behaviour in hbase cluster. It is almost idle, only <5 puts and
gets.
> But the data in hdfs is increasing, and region servers have very high iowait(>100,
in 2 core CPU).
> iotop shows that datanode process is reading and writing all the time.
> Any suggestions?
> 
> Thanks.
> 


Mime
View raw message