hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Basu,Indrashish" <indrash...@ufl.edu>
Subject Re: Error putting files in the HDFS
Date Tue, 08 Oct 2013 20:46:04 GMT
  

Hi Tariq, 

Thanks for your help again. 

I tried deleting the old
HDFS files and directories as you suggested , and then do the
reformatting and starting all the nodes. However after running the
dfsadmin report I am again seeing that datanode is not generated.


root@tegra-ubuntu:~/hadoop-gpu-master/hadoop-gpu-0.20.1# bin/hadoop
dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0
KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: �%
Under
replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks:
0

-------------------------------------------------
Datanodes
available: 0 (0 total, 0 dead) 

However when I typed jps, it is showing
that datanode is up and running. Below are the datanode logs generated
for the given time stamp. Can you kindly assist regarding this ? 


2013-10-08 13:35:55,680 INFO
org.apache.hadoop.hdfs.server.common.Storage: Storage directory
/app/hadoop/tmp/dfs/data is not formatted.
2013-10-08 13:35:55,680 INFO
org.apache.hadoop.hdfs.server.common.Storage: Formatting ...
2013-10-08
13:35:55,814 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Registered FSDatasetStatusMBean
2013-10-08 13:35:55,820 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at
50010
2013-10-08 13:35:55,828 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is
1048576 bytes/s
2013-10-08 13:35:56,153 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2013-10-08 13:35:56,497 INFO
org.apache.hadoop.http.HttpServer: Port returned by
webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening
the listener on 50075
2013-10-08 13:35:56,498 INFO
org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned
50075 webServer.getConnectors()[0].getLocalPort() returned
50075
2013-10-08 13:35:56,513 INFO org.apache.hadoop.http.HttpServer:
Jetty bound to port 50075
2013-10-08 13:35:56,514 INFO org.mortbay.log:
jetty-6.1.14
2013-10-08 13:40:45,127 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50075
2013-10-08 13:40:45,139 INFO
org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with
processName=DataNode, sessionId=null
2013-10-08 13:40:45,189 INFO
org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with
hostName=DataNode, port=50020
2013-10-08 13:40:45,198 INFO
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-10-08
13:40:45,201 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on
50020: starting
2013-10-08 13:40:45,201 INFO
org.apache.hadoop.ipc.Server: IPC Server listener on 50020:
starting
2013-10-08 13:40:45,202 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(tegra-ubuntu:50010, storageID=, infoPort=50075,
ipcPort=50020)
2013-10-08 13:40:45,206 INFO
org.apache.hadoop.ipc.Server: IPC Server handler 2 on 50020:
starting
2013-10-08 13:40:45,207 INFO org.apache.hadoop.ipc.Server: IPC
Server handler 1 on 50020: starting
2013-10-08 13:40:45,234 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: New storage id
DS-863644283-127.0.1.1-50010-1381264845208 is assigned to data-node
10.227.56.195:50010
2013-10-08 13:40:45,235 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(10.227.56.195:50010,
storageID=DS-863644283-127.0.1.1-50010-1381264845208, infoPort=50075,
ipcPort=50020)In DataNode.run, data =
FSDataset{
dirpath='/app/hadoop/tmp/dfs/data/current'}
2013-10-08
13:40:45,235 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: using
BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 0msec
2013-10-08
13:40:45,275 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
BlockReport of 0 blocks got processed in 14 msecs
2013-10-08
13:40:45,277 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Starting Periodic block scanner.

Regards, 

Indrashish 

On Wed, 9 Oct
2013 01:38:52 +0530, Mohammad Tariq wrote: 

> You are welcome Basu. 
>
Not a problem. You can use BIN/HADOOP FS -LSR / to list down all the
HDFS files and directories. See which files are no longer required and
delete them using BIN/HADOOP FS -RM /PATH/TO/THE/FILE 
> 
> Warm
Regards, 
> Tariq 
> cloudfront.blogspot.com [8] 
> 
> On Tue, Oct 8,
2013 at 11:59 PM, Basu,Indrashish wrote:
> 
>> Hi Tariq, 
>> 
>> Thanks
a lot for your help. 
>> 
>> Can you please let me know the path where I
can check the old files in the HDFS and remove them accordingly. I am
sorry to bother with these questions, I am absolutely new to Hadoop. 
>>

>> Thanks again for your time and pateince. 
>> 
>> Regards, 
>> 
>>
Indrashish 
>> 
>> On Tue, 8 Oct 2013 23:51:30 +0530, Mohammad Tariq
wrote: 
>> 
>>> You don't have any more space left in your HDFS. Delete
some old data or add additional storage. 
>>> 
>>> Warm Regards, 
>>>
Tariq 
>>> cloudfront.blogspot.com [6] 
>>> 
>>> On Tue, Oct 8, 2013 at
11:47 PM, Basu,Indrashish wrote:
>>> 
>>>> Hi ,
>>>> 
>>>> Just to
update on this, I have deleted all the old logs and files from the /tmp
and /app/hadoop directory, and restarted all the nodes, I have now 1
datanode available as per the below information :
>>>> 
>>>> Configured
Capacity: 3665985536 (3.41 GB)
>>>> Present Capacity: 24576 (24 KB)

>>>> 
>>>> DFS Remaining: 0 (0 KB) DFS Used: 24576 (24 KB)
>>>> DFS
Used%: 100% 
>>>> 
>>>> Under replicated blocks: 0
>>>> Blocks with
corrupt replicas: 0
>>>> Missing blocks: 0
>>>> 
>>>>
------------------------------------------------- Datanodes available: 1
(1 total, 0 dead)
>>>> 
>>>> Name: 10.227.56.195:50010 [5]
>>>>
Decommission Status : Normal
>>>> Configured Capacity: 3665985536 (3.41
GB)
>>>> DFS Used: 24576 (24 KB)
>>>> Non DFS Used: 3665960960 (3.41
GB)
>>>> DFS Remaining: 0(0 KB)
>>>> DFS Used%: 0%
>>>> DFS Remaining%:
0%
>>>> Last contact: Tue Oct 08 11:12:19 PDT 2013
>>>> 
>>>> However
when I tried putting the files back in HDFS, I am getting the same error
as stated earlier. Do I need to clear some space for the HDFS ?
>>>>

>>>> Regards,
>>>> Indrashish 
>>>> 
>>>> On Tue, 08 Oct 2013 14:01:19
-0400, Basu,Indrashish wrote:
>>>> 
>>>>> Hi Jitendra,
>>>>> 
>>>>> This
is what I am getting in the datanode logs :
>>>>> 
>>>>> 2013-10-07
11:27:41,960 INFO
>>>>> org.apache.hadoop.hdfs.server.common.Storage:
Storage directory
>>>>> /app/hadoop/tmp/dfs/data is not formatted.
>>>>>
2013-10-07 11:27:41,961 INFO
>>>>>
org.apache.hadoop.hdfs.server.common.Storage: Formatting ...
>>>>>
2013-10-07 11:27:42,094 INFO
>>>>>
org.apache.hadoop.hdfs.server.datanode.DataNode: Registered
>>>>>
FSDatasetStatusMBean
>>>>> 2013-10-07 11:27:42,099 INFO
>>>>>
org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server
at
>>>>> 50010
>>>>> 2013-10-07 11:27:42,107 INFO
>>>>>
org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith
is
>>>>> 1048576 bytes/s
>>>>> 2013-10-07 11:27:42,369 INFO
org.mortbay.log: Logging to
>>>>>
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>>>>
org.mortbay.log.Slf4jLog
>>>>> 2013-10-07 11:27:42,632 INFO
org.apache.hadoop.http.HttpServer: Port
>>>>> returned by
webServer.getConnectors()[0].getLocalPort() before open()
>>>>> is -1.
Opening the listener on 50075
>>>>> 2013-10-07 11:27:42,633 INFO
org.apache.hadoop.http.HttpServer:
>>>>> listener.getLocalPort()
returned 50075
>>>>> webServer.getConnectors()[0].getLocalPort()
returned 50075
>>>>> 2013-10-07 11:27:42,634 INFO
org.apache.hadoop.http.HttpServer: Jetty
>>>>> bound to port 50075
>>>>>
2013-10-07 11:27:42,634 INFO org.mortbay.log: jetty-6.1.14
>>>>>
2013-10-07 11:31:29,821 INFO org.mortbay.log: Started
>>>>>
SelectChannelConnector@0.0.0.0:50075 [2]
>>>>> 2013-10-07 11:31:29,843
INFO
>>>>> org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM
Metrics
>>>>> with processName=DataNode, sessionId=null
>>>>> 2013-10-07
11:31:29,912 INFO
>>>>> org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics
>>>>> with hostName=DataNode, port=50020
>>>>>
2013-10-07 11:31:29,922 INFO org.apache.hadoop.ipc.Server: IPC
Server
>>>>> Responder: starting
>>>>> 2013-10-07 11:31:29,922 INFO
org.apache.hadoop.ipc.Server: IPC Server
>>>>> listener on 50020:
starting
>>>>> 2013-10-07 11:31:29,933 INFO
org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 0 on 50020:
starting
>>>>> 2013-10-07 11:31:29,933 INFO
org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 1 on 50020:
starting
>>>>> 2013-10-07 11:31:29,933 INFO
org.apache.hadoop.ipc.Server: IPC Server
>>>>> handler 2 on 50020:
starting
>>>>> 2013-10-07 11:31:29,934 INFO
>>>>>
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
>>>>>
DatanodeRegistration(tegra-ubuntu:50010, storageID=,
infoPort=50075,
>>>>> ipcPort=50020)
>>>>> 2013-10-07 11:31:29,971
INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: New storage
id
>>>>> DS-1027334635-127.0.1.1-50010-1381170689938 is assigned to
data-node
>>>>> 10.227.56.195:50010 [3]
>>>>> 2013-10-07 11:31:29,973
INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>>>>
DatanodeRegistration(10.227.56.195:50010 [4],
>>>>>
storageID=DS-1027334635-127.0.1.1-50010-1381170689938,
infoPort=50075,
>>>>> ipcPort=50020)In DataNode.run, data =
FSDataset
>>>>> {dirpath='/app/hadoop/tmp/dfs/data/current'}
>>>>>
2013-10-07 11:31:29,974 INFO
>>>>>
org.apache.hadoop.hdfs.server.datanode.DataNode: using
>>>>>
BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 0msec
>>>>>
2013-10-07 11:31:30,032 INFO
>>>>>
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0
>>>>>
blocks got processed in 19 msecs
>>>>> 2013-10-07 11:31:30,035
INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Starting
Periodic
>>>>> block scanner.
>>>>> 2013-10-07 11:41:42,222 INFO
>>>>>
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0
>>>>>
blocks got processed in 20 msecs
>>>>> 2013-10-07 12:41:43,482
INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport
of 0
>>>>> blocks got processed in 22 msecs
>>>>> 2013-10-07
13:41:44,755 INFO
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
BlockReport of 0
>>>>> blocks got processed in 13 msecs
>>>>> 
>>>>> I
restarted the datanode and made sure that it is up and running
>>>>>
(typed jps command).
>>>>> 
>>>>> Regards,
>>>>> Indrashish
>>>>> 
>>>>>
On Tue, 8 Oct 2013 23:25:25 +0530, Jitendra Yadav wrote:
>>>>> 
>>>>>>
As per your dfs report, available DataNodes count is ZERO in you
cluster.
>>>>>> 
>>>>>> Please check your data node logs.
>>>>>> 
>>>>>>
Regards
>>>>>> Jitendra
>>>>>> 
>>>>>> On 10/8/13, Basu,Indrashish
wrote:
>>>>>> 
>>>>>>> Hello,
>>>>>>> 
>>>>>>> My name is Indrashish
Basu and I am a Masters student in the Department
>>>>>>> of Electrical
and Computer Engineering.
>>>>>>> 
>>>>>>> Currently I am doing my
research project on Hadoop implementation on
>>>>>>> ARM processor and
facing an issue while trying to run a sample Hadoop
>>>>>>> source code
on the same. Every time I am trying to put some files in the
>>>>>>>
HDFS, I am getting the below error.
>>>>>>> 
>>>>>>> 13/10/07 11:31:29
WARN hdfs.DFSClient: DataStreamer Exception:
>>>>>>>
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>>>>>
/user/root/bin/cpu-kmeans2D could only be replicated to 0 nodes,
instead
>>>>>>> of 1
>>>>>>> at
>>>>>>> 
>>>>>>>
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
>>>>>>>
at
>>>>>>> 
>>>>>>>
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>>>>>>>
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>
at
>>>>>>> 
>>>>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>
at
>>>>>>> 
>>>>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>
at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>>>>>>> at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>>>>>>> at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>>>>>>> at
java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at
javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>> at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>>>>>>

>>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:739)
>>>>>>>
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>>>>>>> at
com.sun.proxy.$Proxy0.addBlock(Unknown Source)
>>>>>>> at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>
at
>>>>>>> 
>>>>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>
at
>>>>>>> 
>>>>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>
at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at
>>>>>>>

>>>>>>>
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>>>>>
at
>>>>>>> 
>>>>>>>
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>>>>>
at com.sun.proxy.$Proxy0.addBlock(Unknown Source)
>>>>>>> at
>>>>>>>

>>>>>>>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
>>>>>>>
at
>>>>>>> 
>>>>>>>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
>>>>>>>
at
>>>>>>> 
>>>>>>>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
>>>>>>>
at
>>>>>>> 
>>>>>>>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>>>>>>>

>>>>>>> 13/10/07 11:31:29 WARN hdfs.DFSClient: Error Recovery for
block null
>>>>>>> bad datanode[0] nodes == null
>>>>>>> 13/10/07
11:31:29 WARN hdfs.DFSClient: Could not get block locations.
>>>>>>>
Source file "/user/root/bin/cpu-kmeans2D" - Aborting...
>>>>>>> put:
java.io.IOException: File /user/root/bin/cpu-kmeans2D could only
>>>>>>>
be replicated to 0 nodes, instead of 1
>>>>>>> 
>>>>>>> I tried
replicating the namenode and datanode by deleting all the old
>>>>>>>
logs on the master and the slave nodes as well as the folders
under
>>>>>>> /app/hadoop/, after which I formatted the namenode and
started the
>>>>>>> process again (bin/start-all.sh), but still no luck
with the same.
>>>>>>> 
>>>>>>> I tried generating the admin
report(pasted below) after doing the
>>>>>>> restart, it seems the data
node is not getting started.
>>>>>>> 
>>>>>>>
-------------------------------------------------
>>>>>>> Datanodes
available: 0 (0 total, 0 dead)
>>>>>>> 
>>>>>>>
root@tegra-ubuntu:~/hadoop-gpu-master/hadoop-gpu-0.20.1#
bin/hadoop
>>>>>>> dfsadmin -report
>>>>>>> Configured Capacity: 0 (0
KB)
>>>>>>> Present Capacity: 0 (0 KB)
>>>>>>> DFS Remaining: 0 (0
KB)
>>>>>>> DFS Used: 0 (0 KB)
>>>>>>> DFS Used%: �%
>>>>>>> Under
replicated blocks: 0
>>>>>>> Blocks with corrupt replicas: 0
>>>>>>>
Missing blocks: 0
>>>>>>> 
>>>>>>>
-------------------------------------------------
>>>>>>> Datanodes
available: 0 (0 total, 0 dead)
>>>>>>> 
>>>>>>> I have tried the
following methods to debug the process :
>>>>>>> 
>>>>>>> 1) I logged in
to the HADOOP home directory and removed all the old
>>>>>>> logs (rm
-rf logs/*)
>>>>>>> 
>>>>>>> 2) Next I deleted the contents of the
directory on all my slave and
>>>>>>> master nodes (rm -rf
/app/hadoop/*)
>>>>>>> 
>>>>>>> 3) I formatted the namenode (bin/hadoop
namenode -format)
>>>>>>> 
>>>>>>> 4) I started all the processes -
first the namenode, datanode and then
>>>>>>> the map - reduce. I typed
jps on the terminal to ensure that all the
>>>>>>> processes (Namenode,
Datanode, JobTracker, Task Tracker) are up and
>>>>>>> running.
>>>>>>>

>>>>>>> 5) Now doing this, I recreated the directories in the
dfs.
>>>>>>> 
>>>>>>> However still no luck with the process.
>>>>>>>

>>>>>>> Can you kindly assist regarding this ? I am new to Hadoop and
I
am
>>>>>>> having no idea as how I can proceed with this.
>>>>>>>

>>>>>>> Regards,
>>>>>>> 
>>>>>>> --
>>>>>>> Indrashish Basu
>>>>>>>
Graduate Student
>>>>>>> Department of Electrical and Computer
Engineering
>>>>>>> University of Florida
>>>> 
>>>> -- 
>>>> Indrashish
Basu
>>>> Graduate Student
>>>> Department of Electrical and Computer
Engineering
>>>> University of Florida
>> 
>> -- 
>> 
>> Indrashish Basu

>> Graduate Student 
>> Department of Electrical and Computer
Engineering 
>> University of Florida

-- 

Indrashish Basu 
Graduate
Student 
Department of Electrical and Computer Engineering 
University
of Florida

  

Links:
------
[1] mailto:indrashish@ufl.edu
[2]
http://SelectChannelConnector@0.0.0.0:50075
[3]
http://10.227.56.195:50010
[4] http://10.227.56.195:50010
[5]
http://10.227.56.195:50010
[6] http://cloudfront.blogspot.com
[7]
mailto:indrashish@ufl.edu
[8] http://cloudfront.blogspot.com
[9]
mailto:indrashish@ufl.edu

Mime
View raw message