hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vigith Maurice (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3333) java.io.IOException: File /user/root/lwr/test31.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
Date Sun, 29 Apr 2012 20:31:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264597#comment-13264597
] 

Vigith Maurice commented on HDFS-3333:
--------------------------------------

Hi,

I had a similar problem, but the reason was because port was not opened between the client
host and the datanode. since port was not open, the connection was getting timed out and ended
up blacklisting each datanode. (i was using amazon ec2 instances where ports are not open
by default.)
                
> java.io.IOException: File /user/root/lwr/test31.txt could only be replicated to 0 nodes
instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded
in this operation.
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3333
>                 URL: https://issues.apache.org/jira/browse/HDFS-3333
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.23.1, 2.0.0
>         Environment: namenodeļ¼š1 (IP:10.18.40.154)
> datanodeļ¼š3 (IP:10.18.40.154,10.18.40.102,10.18.52.55)
> HOST-10-18-40-154:/home/APril20/install/hadoop/namenode/bin # ./hadoop dfsadmin -report
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
> Instead use the hdfs command for it.
> Configured Capacity: 129238446080 (120.36 GB)
> Present Capacity: 51742765056 (48.19 GB)
> DFS Remaining: 49548591104 (46.15 GB)
> DFS Used: 2194173952 (2.04 GB)
> DFS Used%: 4.24%
> Under replicated blocks: 14831
> Blocks with corrupt replicas: 1
> Missing blocks: 100
> -------------------------------------------------
> Datanodes available: 3 (3 total, 0 dead)
> Live datanodes:
> Name: 10.18.40.102:50010 (10.18.40.102)
> Hostname: linux.site
> Decommission Status : Normal
> Configured Capacity: 22765834240 (21.2 GB)
> DFS Used: 634748928 (605.34 MB)
> Non DFS Used: 1762299904 (1.64 GB)
> DFS Remaining: 20368785408 (18.97 GB)
> DFS Used%: 2.79%
> DFS Remaining%: 89.47%
> Last contact: Fri Apr 27 10:35:57 IST 2012
> Name: 10.18.40.154:50010 (HOST-10-18-40-154)
> Hostname: HOST-10-18-40-154
> Decommission Status : Normal
> Configured Capacity: 23259897856 (21.66 GB)
> DFS Used: 812396544 (774.76 MB)
> Non DFS Used: 8297279488 (7.73 GB)
> DFS Remaining: 14150221824 (13.18 GB)
> DFS Used%: 3.49%
> DFS Remaining%: 60.84%
> Last contact: Fri Apr 27 10:35:58 IST 2012
> Name: 10.18.52.55:50010 (10.18.52.55)
> Hostname: HOST-10-18-52-55
> Decommission Status : Normal
> Configured Capacity: 83212713984 (77.5 GB)
> DFS Used: 747028480 (712.42 MB)
> Non DFS Used: 67436101632 (62.8 GB)
> DFS Remaining: 15029583872 (14 GB)
> DFS Used%: 0.9%
> DFS Remaining%: 18.06%
> Last contact: Fri Apr 27 10:35:58 IST 2012
>            Reporter: liaowenrui
>   Original Estimate: 0.2h
>  Remaining Estimate: 0.2h
>
> log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> java.io.IOException: File /user/root/lwr/test31.txt could only be replicated to 0 nodes
instead of minReplication (=1).  There are 3 datanode(s) running and 3 node(s) are excluded
in this operation.
> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1259)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1916)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:472)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292)
> 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:428)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:905)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1684)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1205)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1682)
> i:4284
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1159)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:185)
> 	at $Proxy9.addBlock(Unknown Source)
> 	at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
> 	at $Proxy9.addBlock(Unknown Source)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:295)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1097)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:973)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:455)
> testcase:
> import java.io.IOException;
> import java.net.URI;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FSDataOutputStream;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.hdfs.DFSConfigKeys;
> import org.apache.hadoop.hdfs.DistributedFileSystem;
> public class Write1 {
> 	/**
> 	 * @param args
> 	 * @throws Exception 
> 	 */
> 	public static void main(String[] args) throws Exception {
> 		//System.out.println("main");
> 		String hdfsFile="/user/root/lwr/test31.txt";
>     	byte writeBuff[] = new byte [1024 * 1024];
>     	int i=0;
>     	DistributedFileSystem dfs = new DistributedFileSystem();
>     	Configuration conf=new Configuration();
>     	//conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 512);
>     	//conf.setLong(DFSConfigKeys.DFS_REPLICATION_KEY, 2);
>         // conf.setInt("dfs.replication", 3);
>         conf.setLong("dfs.blocksize", 512);
>     	dfs.initialize(URI.create("hdfs://10.18.40.154:9000"), conf);
>     	//dfs.delete(new Path(hdfsFile));
> 	    //appendFile(dfs,hdfsFile,1024 * 1024,true);
>     	try
>     	{
>     	FSDataOutputStream out1=dfs.create(new Path(hdfsFile));
>     	
>    	    for(i=0;i<100000;i++)
>    	    {
>     	 out1.write(writeBuff, 0, 512);
>     	}
>         out1.hsync();
>         out1.close();
>         /*
> 	    FSDataOutputStream out=dfs.append(new Path(hdfsFile),4096);
> 		out.write(writeBuff, 0, 512 * 1024);
> 		out.hsync();
> 		out.close();
> 		*/
>     	}catch (IOException e)
>     	{    	   
>     	   System.out.println("i:" + i);
>     	   e.printStackTrace();
>     	}
>     	finally
>     	{   
>     		
>     		System.out.println("i:" + i);
>     	    System.out.println("end!");
>     	 
>     	}
>     }
> 	
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message