hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sudhakara st <sudhakara...@gmail.com>
Subject Re: How to set heartbeat ?
Date Thu, 27 Feb 2014 05:41:03 GMT
 Default time interval for marking a datanode as "stale", i.e., if the
namenode has not received heartbeat msg from a datanode for more than this
time interval, the datanode will be marked and treated as "stale" by
default. The stale interval cannot be too small since otherwise this may
cause too frequent change of stale states. We thus set a minimum stale
interval value (the default value is 3 times of heartbeat interval) and
guarantee that the stale interval cannot be less than the minimum value. A
stale data node is avoided during lease/block recovery. It can be
conditionally avoided for reads (see
dfs.namenode.avoid.read.stale.datanode) and for writes (see
dfs.namenode.avoid.write.stale.datanode).
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml


On Thu, Feb 27, 2014 at 10:42 AM, EdwardKing <zhangsc@neusoft.com> wrote:

>  I set hdfs-site.xml,like follows:
>
> <configuration>
> <property>
>  <name>dfs.name.dir</name>
>  <value>file:/home/software/name</value>
>  <description> </description>
> </property>
> <property>
>  <name>dfs.namenode.secondary.http-address</name>
>  <value>master:9001</value>
> </property>
> <property>
>  <name>dfs.data.dir</name>
>  <value>file:/home/software/data</value>
> </property>
> <property>
>  <name>dfs.http.address</name>
>  <value>master:9002</value>
> </property>
> <property>
>  <name>dfs.replication</name>
>  <value>2</value>
> </property>
> <property>
>  <name>dfs.datanode.du.reserved</name>
>  <value>1073741824</value>
> </property>
> <property>
>  <name>dfs.block.size</name>
>  <value>4194304</value>
> </property>
> <property>
>  <name>dfs.namenode.logging.level</name>
>  <value>all</value>
> </property>
> <property>
> <name>dfs.permissions</name>
> <value>false</value>
> </property>
> <property>
>  <name>dfs.namenode.stale.datanode.interval</name>
>  <value>3</value>
> </property>
> </configuration>
>
> Then I check status,like follows:
> [hadoop@master sbin]$ hadoop dfsadmin -report
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
> Instead use the hdfs command for it.
>
> 14/02/26 21:05:11 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Configured Capacity: 34678300672 (32.30 GB)
> Present Capacity: 27971788800 (26.05 GB)
> DFS Remaining: 27657437184 (25.76 GB)
> DFS Used: 314351616 (299.79 MB)
> DFS Used%: 1.12%
> Under replicated blocks: 12
> Blocks with corrupt replicas: 2
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 2 (2 total, 0 dead)
>
> Live datanodes:
> Name: 172.11.12.7:50010 (node1)
> Hostname: node1
> Decommission Status : Normal
> Configured Capacity: 17339150336 (16.15 GB)
> DFS Used: 157192192 (149.91 MB)
> Non DFS Used: 2949230592 (2.75 GB)
> DFS Remaining: 14232727552 (13.26 GB)
> DFS Used%: 0.91%
> DFS Remaining%: 82.08%
> Last contact: Wed Feb 26 21:02:22 PST 2014
>
>
> Name: 172.11.12.6:50010 (master)
> Hostname: master
> Decommission Status : Normal
> Configured Capacity: 17339150336 (16.15 GB)
> DFS Used: 157159424 (149.88 MB)
> Non DFS Used: 3757281280 (3.50 GB)
> DFS Remaining: 13424709632 (12.50 GB)
> DFS Used%: 0.91%
> DFS Remaining%: 77.42%
> Last contact: Wed Feb 26 21:05:13 PST 2014
>
> Then I kill slave DataNode,like follows:
>
> [hadoop@node1 ~]$ jps
> 4529 DataNode
> 4805 Jps
> [hadoop@node1 ~]$ kill -9 4529
> [hadoop@node1 ~]$ jps
> 4817 Jps
>
> After a very long time,I check the status again
> [hadoop@master hadoop]$ hadoop dfsadmin -report
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
> Instead use the hdfs command for it.
>
> 14/02/26 21:08:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Configured Capacity: 34678300672 (32.30 GB)
> Present Capacity: 27971780608 (26.05 GB)
> DFS Remaining: 27657428992 (25.76 GB)
> DFS Used: 314351616 (299.79 MB)
> DFS Used%: 1.12%
> Under replicated blocks: 12
> Blocks with corrupt replicas: 2
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 2 (2 total, 0 dead)
>
> Live datanodes:
> Name: 172.11.12.7:50010 (node1)
> Hostname: node1
> Decommission Status : Normal
> Configured Capacity: 17339150336 (16.15 GB)
> DFS Used: 157192192 (149.91 MB)
> Non DFS Used: 2949230592 (2.75 GB)
> DFS Remaining: 14232727552 (13.26 GB)
> DFS Used%: 0.91%
> DFS Remaining%: 82.08%
> Last contact: Wed Feb 26 21:02:22 PST 2014
>
>
> Name: 172.11.12.6:50010 (master)
> Hostname: master
> Decommission Status : Normal
> Configured Capacity: 17339150336 (16.15 GB)
> DFS Used: 157159424 (149.88 MB)
> Non DFS Used: 3757289472 (3.50 GB)
> DFS Remaining: 13424701440 (12.50 GB)
> DFS Used%: 0.91%
> DFS Remaining%: 77.42%
> Last contact: Wed Feb 26 21:08:22 PST 2014
>
>
> Why can't  I get  slave node dead?  Where is wrong? Thanks
>
>
> ----- Original Message -----
> *From:* sudhakara st <sudhakara.st@gmail.com>
> *To:* user@hadoop.apache.org
> *Sent:* Thursday, February 27, 2014 11:36 AM
> *Subject:* Re: How to set heartbeat ?
>
>  In hdfs-site.xml *'dfs.heartbeat.interval*' parameter determines
> datanode heartbeat interval in seconds.
> and have look these variables this parameter also
> *'dfs.namenode.stale.datanode.interval'*
>
>
> On Thu, Feb 27, 2014 at 7:08 AM, EdwardKing <zhangsc@neusoft.com> wrote:
>
>>  I have two machine,one is master and another is slave, I want to know
>> how to configure heartbeat of hadoop 2.2.0,which file will be modified?
>> Thanks.
>>
>>
>> ---------------------------------------------------------------------------------------------------
>> Confidentiality Notice: The information contained in this e-mail and any
>> accompanying attachment(s)
>> is intended only for the use of the intended recipient and may be
>> confidential and/or privileged of
>> Neusoft Corporation, its subsidiaries and/or its affiliates. If any
>> reader of this communication is
>> not the intended recipient, unauthorized use, forwarding, printing,
>> storing, disclosure or copying
>> is strictly prohibited, and may be unlawful.If you have received this
>> communication in error,please
>> immediately notify the sender by return e-mail, and delete the original
>> message and all copies from
>> your system. Thank you.
>>
>> ---------------------------------------------------------------------------------------------------
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
>
> ---------------------------------------------------------------------------------------------------
>



-- 

Regards,
...sudhakara

Mime
View raw message