hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: High iowait in idle hbase cluster
Date Wed, 02 Sep 2015 16:45:25 GMT
bq. change the ip addresses of the cluster nodes

Did this happen recently ? If high iowait was observed after the change
(you can look at ganglia graph), there is a chance that the change was
related.

BTW I assume 10.10.8.55 <http://10.10.8.55:50010/> is where your region
server resides.

Cheers

On Wed, Sep 2, 2015 at 9:39 AM, Akmal Abbasov <akmal.abbasov@icloud.com>
wrote:

> Hi Ted,
> sorry forget to mention
>
> release of hbase / hadoop you're using
>
> hbase hbase-0.98.7-hadoop2, hadoop hadoop-2.5.1
>
> were region servers doing compaction ?
>
> I’ve run major compactions manually earlier today, but it seems that they
> already completed, looking at the compactionQueueSize.
>
> have you checked region server logs ?
>
> The logs of datanode is full of this kind of messages
> 2015-09-02 16:37:06,950 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
> 10.10.8.55:50010, dest: /10.10.8.54:32959, bytes: 19673, op: HDFS_READ,
> cliID: DFSClient_NONMAPREDUCE_1225374853_1, offset: 0, srvID:
> ee7d0634-89a3-4ada-a8ad-7848217327be, blockid:
> BP-329084760-10.32.0.180-1387281790961:blk_1075277914_1540222, duration:
> 7881815
>
> p.s. we had to change the ip addresses of the cluster nodes, is it
> relevant?
>
> Thanks.
>
> On 02 Sep 2015, at 18:20, Ted Yu <yuzhihong@gmail.com> wrote:
>
> Please provide some more information:
>
> release of hbase / hadoop you're using
> were region servers doing compaction ?
> have you checked region server logs ?
>
> Thanks
>
> On Wed, Sep 2, 2015 at 9:11 AM, Akmal Abbasov <akmal.abbasov@icloud.com>
> wrote:
>
>> Hi,
>> I’m having strange behaviour in hbase cluster. It is almost idle, only <5
>> puts and gets.
>> But the data in hdfs is increasing, and region servers have very high
>> iowait(>100, in 2 core CPU).
>> iotop shows that datanode process is reading and writing all the time.
>> Any suggestions?
>>
>> Thanks.
>
>
>
>

Mime
View raw message