hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xeon <xeonmailingl...@gmail.com>
Subject Re: There are 2 datanode(s) running and 2 node(s) are excluded in this operation.
Date Wed, 28 Aug 2013 11:42:28 GMT
I am copying from Site1 to Site2 in Amazon EC2. Each site has mounted 1 
hadoop instance with 1 NN and 2 DNs. I think my problem is something 
related distcp command being unable to query one of the HDFS' clusters' 
DNs like Harsh told. Amazon EC2 use private IPs to communicate between 
hosts in the same site and public IPs to connect between sites. I am 
using distcp to copy big data (256MB each file) between different sites 
using the public IP:
hadoop distcp hdfs://publicIP1:9000/wiki hdfs://publicIP2:9000/wiki

Checking the logs (hadoop-ubuntu-namenode-ip-XX-XXX-XXX-94.log), I also 
got this problem  but I don't know if it is related:
2013-08-28 07:40:17,537 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 3 on 9000, call 
org.apache.hadoop.hdfs.protocol.ClientProtocol.abandonBlock from 
XX.XXX.XXX.150:51844: error: 
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease 
on /wiki/.distcp.tmp.attempt_1377674569447_0001_m_000000_1: File does 
not exist. H
older DFSClient_attempt_1377674569447_0001_m_000000_1_-1864394963_1 does 
not have any open files.
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease 
on /wiki/.distcp.tmp.attempt_1377674569447_0001_m_000000_1: File does 
not exist. Holder DFSClient_
attempt_1377674569447_0001_m_000000_1_-1864394963_1 does not have any 
open files.


Here is the report:

Site1
$ hdfs dfsadmin -report
13/08/28 11:17:52 WARN util.NativeCodeLoader: Unable to load 
native-hadoop library for your platform... using builtin-java classes 
where applicable
Configured Capacity: 25365413888 (23.62 GB)
Present Capacity: 20488749056 (19.08 GB)
DFS Remaining: 20487512064 (19.08 GB)
DFS Used: 1236992 (1.18 MB)
DFS Used%: 0.01%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)

Live datanodes:
Name: XX.XXX.XXX.243:50010 (ip-XX-XXX-XXX-243.eu-west-1.compute.internal)
Hostname: ip-XX-XXX-XXX-243.eu-west-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 618496 (604 KB)
Non DFS Used: 2489556992 (2.32 GB)
DFS Remaining: 10192531456 (9.49 GB)
DFS Used%: 0.00%
DFS Remaining%: 80.37%
Last contact: Wed Aug 28 11:17:51 UTC 2013


Name: XX.XXX.XXX.58:50010 (ip-XX-XXX-XXX-58.eu-west-1.compute.internal)
Hostname: ip-XX-XXX-XXX-58.eu-west-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 618496 (604 KB)
Non DFS Used: 2387107840 (2.22 GB)
DFS Remaining: 10294980608 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.17%
Last contact: Wed Aug 28 11:17:51 UTC 2013


Site2:
$ hdfs dfsadmin -report
13/08/28 11:17:55 WARN util.NativeCodeLoader: Unable to load 
native-hadoop library for your platform... using builtin-java classes 
where applicable
Configured Capacity: 25365413888 (23.62 GB)
Present Capacity: 20593844224 (19.18 GB)
DFS Remaining: 20593770496 (19.18 GB)
DFS Used: 73728 (72 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 2 (2 total, 0 dead)

Live datanodes:
Name: XX.XXX.XXX.95:50010 (ip-XX-XXX-XXX-95.ap-southeast-1.compute.internal)
Hostname: ip-XX-XXX-XXX-95.ap-southeast-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 36864 (36 KB)
Non DFS Used: 2385768448 (2.22 GB)
DFS Remaining: 10296901632 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.19%
Last contact: Wed Aug 28 11:17:56 UTC 2013


Name: XX.XXX.XXX.96:50010 (ip-XX-XXX-XXX-96.ap-southeast-1.compute.internal)
Hostname: ip-XX-XXX-XXX-96.ap-southeast-1.compute.internal
Decommission Status : Normal
Configured Capacity: 12682706944 (11.81 GB)
DFS Used: 36864 (36 KB)
Non DFS Used: 2385801216 (2.22 GB)
DFS Remaining: 10296868864 (9.59 GB)
DFS Used%: 0.00%
DFS Remaining%: 81.19%
Last contact: Wed Aug 28 11:17:56 UTC 2013

Any suggestion to fix this problem?






On 08/28/2013 12:09 PM, Jitendra Yadav wrote:
> Hi,
> Also can you please share the dfs heath check report of your cluster?
> Thanks
> On Wed, Aug 28, 2013 at 3:46 PM, xeon <xeonmailinglist@gmail.com 
> <mailto:xeonmailinglist@gmail.com>> wrote:
>
>     Hi,
>
>      I don't have the "dfs.hosts.exclude" property defined, but I
>     still get the error "There are 2 datanode(s) running and 2 node(s)
>     are excluded in this operation." when I run the distcp command.
>     Any help?
>
>


Mime
View raw message