hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Lederman <jon2...@gmail.com>
Subject Re: Lost Task Tracker because of no heartbeat
Date Sun, 20 Mar 2011 15:21:28 GMT
Hi,

I am running into an issue when running Pi Estimator in the examples for Hadoop.  I am able
to format the namenode and start all daemons.

I also cleared the tmp/hadoop-${USER} directory first.

When I launch the job, it crashes with an error about not being able o replicate.  I am attaching
the output.  I am also getting errors about bad datanodes.  I have reinstalled Hadoop and
cleared the tmp directories.  Does anybody have any suggestions about what the problem might
be?

Thanks.

-Jon

-sh-4.1# hadoop jar hadoop-0.20.2-examples.jar pi 2 10
Number of Maps  = 2
Samples per Map = 10
70/01/01 00:07:51 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File /user/root/PiEstimator_TMP_3_141592654/in/part0 could only be replicated
to 0 nodes, instead of 1
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

	at org.apache.hadoop.ipc.Client.call(Client.java:740)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy0.addBlock(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at $Proxy0.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2937)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2819)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)

70/01/01 00:07:51 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes
== null
70/01/01 00:07:51 WARN hdfs.DFSClient: Could not get block locations. Source file "/user/root/PiEstimator_TMP_3_141592654/in/part0"
- Aborting...
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/root/PiEstimator_TMP_3_141592654/in/part0
could only be replicated to 0 nodes, instead of 1
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

	at org.apache.hadoop.ipc.Client.call(Client.java:740)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy0.addBlock(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at $Proxy0.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2937)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2819)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
70/01/01 00:07:51 ERROR hdfs.DFSClient: Exception closing file /user/root/PiEstimator_TMP_3_141592654/in/part0
: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/root/PiEstimator_TMP_3_141592654/in/part0
could only be replicated to 0 nodes, instead of 1
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/root/PiEstimator_TMP_3_141592654/in/part0
could only be replicated to 0 nodes, instead of 1
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

	at org.apache.hadoop.ipc.Client.call(Client.java:740)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy0.addBlock(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at $Proxy0.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2937)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2819)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
-sh-4.1# jps
883 SecondaryNameNode
1264 Jps
734 NameNode
808 DataNode
1029 TaskTracker
946 JobTracker
-sh-4.1# hadoop fs -ls /
Found 2 items
drwxr-xr-x   - root supergroup          0 1970-01-01 00:07 /tmp
drwxr-xr-x   - root supergroup          0 1970-01-01 00:07 /user
-sh-4.1# hadoop fs -ls /tmp
Found 1 items
drwxr-xr-x   - root supergroup          0 1970-01-01 00:07 /tmp/hadoop-root
-sh-4.1# hadoop fs -ls /tmp/hadoop-root
Found 1 items
drwxr-xr-x   - root supergroup          0 1970-01-01 00:07 /tmp/hadoop-root/mapred
-sh-4.1# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             459G  2.8G  433G   1% /
tmpfs                 978M     0  978M   0% /dev/shm
-sh-4.1# hadoop dfsadmin -report
Configured Capacity: 492256313344 (458.45 GB)
Present Capacity: 464298119183 (432.41 GB)
DFS Remaining: 464298094592 (432.41 GB)
DFS Used: 24591 (24.01 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)

Name: 127.0.0.1:50010
Decommission Status : Normal
Configured Capacity: 492256313344 (458.45 GB)
DFS Used: 24591 (24.01 KB)
Non DFS Used: 27958194161 (26.04 GB)
DFS Remaining: 464298094592(432.41 GB)
DFS Used%: 0%
DFS Remaining%: 94.32%
Last contact: Thu Jan 01 00:16:04 GMT 1970
On Mar 16, 2011, at 4:14 PM, Harsh J wrote:

> Hello,
> 
> On Thu, Mar 17, 2011 at 1:39 AM, baran cakici <barancakici@gmail.com> wrote:
>> @Harsh
>> 
>> I start daemons with start-dfs.sh and then start-mapred-dfs.sh. do you mean
>> this Exception(org.apache.hadoop.ipc.RemoteException) is normal?
> 
> Yes. It is additionally logged as INFO. This isn't a problem since NN
> needs to be up before JT can use it.
> 
> -- 
> Harsh J
> http://harshj.com


Mime
View raw message