hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ronen Itkin <ro...@taykey.com>
Subject Re: HDFS DataNode daily log growing really high and fast
Date Mon, 31 Oct 2011 05:59:54 GMT
For instance, yesterday's daily log:
"/var/log/hadoop/hadoop-hadoop-datanode-ip-10-10-10-4.log"
on the problematic Node03 was in the size of 1.1 GB while on other Nodes
the same log was in the size of 87 MB.

Again, nothing is being run specifically on Node03, I have 3 nodes, with
replication of 3 - means that all the data is being saved on every node,
All nodes are connected to the same switch (and on the same subnet) - so no
advantages to Node03 in any Job.

I am being suspicious regarding HBase...

What do you think? any suggestions?

Thank a head!




On Sun, Oct 30, 2011 at 11:02 AM, Ronen Itkin <ronen@taykey.com> wrote:

> Hey all!
>
> I am having an issue with hadoop's daily datanode log growing to + 1.8 GB.
> I have 3 Nodes in my hdfs cluster, all sharing the same configuration
> (including same log4j.properties).
> While running operations and jobs equally (automatically) on whole of the
> nodes, only one of them (data node*03*)is having this issue with the log
> growing high.
>
> /var/log/hadoop/hadoop-hadoop-datanode-ip-10-10-10-4.log
>
> The log does not show any exceptions, just many hdfs operations
> (read+write).
>
> I am currently running *Cloudera* *hadoop-0.20.2-cdh3u1* and this is my
> architecture:
>
> *MasterServer*: NameNode, JobTracker, HBase HMaster (*hbase-0.90.3-cdh3u1*),
> Zookeeper01
> *Node01*: Data Node, TaskTracker, HBase HRegion
> *Node02*: Data Node, TaskTracker, HBase HRegion
> *Node03*: Data Node, TaskTracker, HBase HRegion
> *SecondaryServer*: Secondary NameNode, HBase Backup HMaster, Zookeeper02
> *ServerX*: ZooKeeper03
>
>
> Does any one can think of a good reason, why it happens ? why to a
> specific node? is is related to hbase operations? hdfs block scanner?
>
> Here is a sample of the log file:
>
> 2011-10-30 08:52:27,313 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /*
> 10.10.10.4*:50010, dest: /*10.10.10.4*:43447, bytes: 66564, op:
> HDFS_READ, cliID: DFSClient_hb_rs_ip-*10-10-10-4*.ec2.internal,60020,1318334166605_1318334167243,
> offset: 34500096, srvID: DS-75443592-10.93.67.113-50010-1318335522512,
> blockid: blk_2773771462926694276_25674, duration: 274702
> 2011-10-30 08:52:27,314 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /*
> 10.10.10.4*:50010, dest: /*10.10.10.4*:43448, bytes: 66564, op:
> HDFS_READ, cliID: DFSClient_hb_rs_ip-*10-10-10-4*.ec2.internal,60020,1318334166605_1318334167243,
> offset: 34631168, srvID: DS-75443592-10.93.67.113-50010-1318335522512,
> blockid: blk_2773771462926694276_25674, duration: 236691
>
>
>
> Thanks,
> *Ronen.*
>
>  <http://www.taykey.com/>
>
>


-- 
*
Ronen Itkin*
Taykey | www.taykey.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message