hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Basu,Indrashish" <indrash...@ufl.edu>
Subject Re: Error putting files in the HDFS
Date Tue, 08 Oct 2013 18:01:19 GMT
Hi Jitendra,

This is what I am getting in the datanode logs :

2013-10-07 11:27:41,960 INFO 
org.apache.hadoop.hdfs.server.common.Storage: Storage directory 
/app/hadoop/tmp/dfs/data is not formatted.
2013-10-07 11:27:41,961 INFO 
org.apache.hadoop.hdfs.server.common.Storage: Formatting ...
2013-10-07 11:27:42,094 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: Registered 
FSDatasetStatusMBean
2013-10-07 11:27:42,099 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 
50010
2013-10-07 11:27:42,107 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 
1048576 bytes/s
2013-10-07 11:27:42,369 INFO org.mortbay.log: Logging to 
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via 
org.mortbay.log.Slf4jLog
2013-10-07 11:27:42,632 INFO org.apache.hadoop.http.HttpServer: Port 
returned by webServer.getConnectors()[0].getLocalPort() before open() is 
-1. Opening the listener on 50075
2013-10-07 11:27:42,633 INFO org.apache.hadoop.http.HttpServer: 
listener.getLocalPort() returned 50075 
webServer.getConnectors()[0].getLocalPort() returned 50075
2013-10-07 11:27:42,634 INFO org.apache.hadoop.http.HttpServer: Jetty 
bound to port 50075
2013-10-07 11:27:42,634 INFO org.mortbay.log: jetty-6.1.14
2013-10-07 11:31:29,821 INFO org.mortbay.log: Started 
SelectChannelConnector@0.0.0.0:50075
2013-10-07 11:31:29,843 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: 
Initializing JVM Metrics with processName=DataNode, sessionId=null
2013-10-07 11:31:29,912 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: 
Initializing RPC Metrics with hostName=DataNode, port=50020
2013-10-07 11:31:29,922 INFO org.apache.hadoop.ipc.Server: IPC Server 
Responder: starting
2013-10-07 11:31:29,922 INFO org.apache.hadoop.ipc.Server: IPC Server 
listener on 50020: starting
2013-10-07 11:31:29,933 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 0 on 50020: starting
2013-10-07 11:31:29,933 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 1 on 50020: starting
2013-10-07 11:31:29,933 INFO org.apache.hadoop.ipc.Server: IPC Server 
handler 2 on 50020: starting
2013-10-07 11:31:29,934 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = 
DatanodeRegistration(tegra-ubuntu:50010, storageID=, infoPort=50075, 
ipcPort=50020)
2013-10-07 11:31:29,971 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: New storage id 
DS-1027334635-127.0.1.1-50010-1381170689938 is assigned to data-node 
10.227.56.195:50010
2013-10-07 11:31:29,973 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: 
DatanodeRegistration(10.227.56.195:50010, 
storageID=DS-1027334635-127.0.1.1-50010-1381170689938, infoPort=50075, 
ipcPort=50020)In DataNode.run, data = FSDataset
{dirpath='/app/hadoop/tmp/dfs/data/current'}
2013-10-07 11:31:29,974 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: using 
BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 0msec
2013-10-07 11:31:30,032 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks 
got processed in 19 msecs
2013-10-07 11:31:30,035 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic block 
scanner.
2013-10-07 11:41:42,222 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks 
got processed in 20 msecs
2013-10-07 12:41:43,482 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks 
got processed in 22 msecs
2013-10-07 13:41:44,755 INFO 
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks 
got processed in 13 msecs


I restarted the datanode and made sure that it is up and running (typed 
jps command).

Regards,
Indrashish

On Tue, 8 Oct 2013 23:25:25 +0530, Jitendra Yadav wrote:
> As per your dfs report, available DataNodes  count is ZERO  in you 
> cluster.
>
> Please check your data node logs.
>
> Regards
> Jitendra
>
> On 10/8/13, Basu,Indrashish <indrashish@ufl.edu> wrote:
>>
>> Hello,
>>
>> My name is Indrashish Basu and I am a Masters student in the 
>> Department
>> of Electrical and Computer Engineering.
>>
>> Currently I am doing my research project on Hadoop implementation on
>> ARM processor and facing an issue while trying to run a sample 
>> Hadoop
>> source code on the same. Every time I am trying to put some files in 
>> the
>> HDFS, I am getting the below error.
>>
>>
>> 13/10/07 11:31:29 WARN hdfs.DFSClient: DataStreamer Exception:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> /user/root/bin/cpu-kmeans2D could only be replicated to 0 nodes, 
>> instead
>> of 1
>> at
>> 
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>> at
>> 
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:739)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>> at com.sun.proxy.$Proxy0.addBlock(Unknown Source)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> 
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> at
>> 
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> at com.sun.proxy.$Proxy0.addBlock(Unknown Source)
>> at
>> 
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
>> at
>> 
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
>> at
>> 
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
>> at
>> 
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>>
>> 13/10/07 11:31:29 WARN hdfs.DFSClient: Error Recovery for block null
>> bad datanode[0] nodes == null
>> 13/10/07 11:31:29 WARN hdfs.DFSClient: Could not get block 
>> locations.
>> Source file "/user/root/bin/cpu-kmeans2D" - Aborting...
>> put: java.io.IOException: File /user/root/bin/cpu-kmeans2D could 
>> only
>> be replicated to 0 nodes, instead of 1
>>
>>
>> I tried replicating the namenode and datanode by deleting all the 
>> old
>> logs on the master and the slave nodes as well as the folders under
>> /app/hadoop/, after which I formatted the namenode and started the
>> process again (bin/start-all.sh), but still no luck with the same.
>>
>> I tried generating the admin report(pasted below) after doing the
>> restart, it seems the data node is not getting started.
>>
>> -------------------------------------------------
>> Datanodes available: 0 (0 total, 0 dead)
>>
>> root@tegra-ubuntu:~/hadoop-gpu-master/hadoop-gpu-0.20.1# bin/hadoop
>> dfsadmin -report
>> Configured Capacity: 0 (0 KB)
>> Present Capacity: 0 (0 KB)
>> DFS Remaining: 0 (0 KB)
>> DFS Used: 0 (0 KB)
>> DFS Used%: �%
>> Under replicated blocks: 0
>> Blocks with corrupt replicas: 0
>> Missing blocks: 0
>>
>> -------------------------------------------------
>> Datanodes available: 0 (0 total, 0 dead)
>>
>>
>> I have tried the following methods to debug the process :
>>
>> 1) I logged in to the HADOOP home directory and removed all the old
>> logs (rm -rf logs/*)
>>
>> 2) Next I deleted the contents of the directory on all my slave and
>> master nodes (rm -rf /app/hadoop/*)
>>
>> 3) I formatted the namenode (bin/hadoop namenode -format)
>>
>> 4) I started all the processes - first the namenode, datanode and 
>> then
>> the map - reduce. I typed jps on the terminal to ensure that all the
>> processes (Namenode, Datanode, JobTracker, Task Tracker) are up and
>> running.
>>
>> 5) Now doing this, I recreated the directories in the dfs.
>>
>> However still no luck with the process.
>>
>>
>> Can you kindly assist regarding this ? I am new to Hadoop and I am
>> having no idea as how I can proceed with this.
>>
>>
>>
>>
>> Regards,
>>
>> --
>> Indrashish Basu
>> Graduate Student
>> Department of Electrical and Computer Engineering
>> University of Florida
>>

-- 
Indrashish Basu
Graduate Student
Department of Electrical and Computer Engineering
University of Florida

Mime
View raw message