hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akmal Abbasov <akmal.abba...@icloud.com>
Subject Re: High iowait in idle hbase cluster
Date Wed, 02 Sep 2015 19:11:05 GMT
Hi Ted,
I’ve checked the time when addresses were changed, and this strange behaviour started weeks
before it.

yes, 10.10.8.55 is region server and 10.10.8.54 is a hbase master.
any thoughts?

Thanks

> On 02 Sep 2015, at 18:45, Ted Yu <yuzhihong@gmail.com> wrote:
> 
> bq. change the ip addresses of the cluster nodes
> 
> Did this happen recently ? If high iowait was observed after the change (you can look
at ganglia graph), there is a chance that the change was related.
> 
> BTW I assume 10.10.8.55 <http://10.10.8.55:50010/> is where your region server
resides.
> 
> Cheers
> 
> On Wed, Sep 2, 2015 at 9:39 AM, Akmal Abbasov <akmal.abbasov@icloud.com <mailto:akmal.abbasov@icloud.com>>
wrote:
> Hi Ted,
> sorry forget to mention
> 
>> release of hbase / hadoop you're using
> 
> hbase hbase-0.98.7-hadoop2, hadoop hadoop-2.5.1
> 
>> were region servers doing compaction ?
> 
> I’ve run major compactions manually earlier today, but it seems that they already completed,
looking at the compactionQueueSize.
> 
>> have you checked region server logs ?
> The logs of datanode is full of this kind of messages
> 2015-09-02 16:37:06,950 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.10.8.55:50010 <http://10.10.8.55:50010/>, dest: /10.10.8.54:32959 <http://10.10.8.54:32959/>,
bytes: 19673, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_1225374853_1, offset: 0, srvID:
ee7d0634-89a3-4ada-a8ad-7848217327be, blockid: BP-329084760-10.32.0.180-1387281790961:blk_1075277914_1540222,
duration: 7881815
> 
> p.s. we had to change the ip addresses of the cluster nodes, is it relevant?
> 
> Thanks.
> 
>> On 02 Sep 2015, at 18:20, Ted Yu <yuzhihong@gmail.com <mailto:yuzhihong@gmail.com>>
wrote:
>> 
>> Please provide some more information:
>> 
>> release of hbase / hadoop you're using
>> were region servers doing compaction ?
>> have you checked region server logs ?
>> 
>> Thanks
>> 
>> On Wed, Sep 2, 2015 at 9:11 AM, Akmal Abbasov <akmal.abbasov@icloud.com <mailto:akmal.abbasov@icloud.com>>
wrote:
>> Hi,
>> I’m having strange behaviour in hbase cluster. It is almost idle, only <5 puts
and gets.
>> But the data in hdfs is increasing, and region servers have very high iowait(>100,
in 2 core CPU).
>> iotop shows that datanode process is reading and writing all the time.
>> Any suggestions?
>> 
>> Thanks.
>> 
> 
> 


Mime
View raw message