hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Swenson <e...@swenson.org>
Subject HDFS Replication Issue
Date Tue, 04 Oct 2016 18:58:54 GMT
I have set up a single node cluster (initially) and am attempting to
write a file from a client outside the cluster.  I’m using the Java
org.apache.hadoop.fs.FileSystem interface to write the file.

While the write call returns, the close call hangs for a very long time, eventually
returns, but the resulting file in HDFS is 0 bytes in length. The namenode log
says:

2016-10-03 22:01:41,367 INFO BlockStateChange: chooseUnderReplicatedBlocks selected 1 blocks
at priority level 0;  Total=1 Reset bookmarks? true
2016-10-03 22:01:41,367 INFO BlockStateChange: BLOCK* neededReplications = 1, pendingReplications
= 0.
2016-10-03 22:01:41,367 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Blocks
chosen but could not be replicated = 1; of which 1 have no target, 0 have no source, 0 are
UC, 0 are abandoned, 0 already have enough replicas.

Why is the block not written to the single datanode (same as
namenode)? What does it mean to "have no target"? The replication
count is 1 and I would have thought that a single copy of the file
would be stored on the single cluster node.

I decided to see what happened if I added a second node to the cluster.  Essentially the same
thing happens.  The file (in HDFS) ends up being zero-length, and I get similar messages from
the NameNode telling me that there are additional neededReplications and that none of the
blocks could be replicated because they “have no target”.

If I SSH into the combined Name/Data node instance and use the “hdfs dfs -put” command,
I have no trouble storing files.  I’m using the same user regardless of whether I’m using
a remote fs.write operation or whether I’m using the “hdfs dfs -put” command while logged
into the NameNode.  

What am I doing wrong?  — Eric




---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message