hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kale, Chetan" <Chetan.K...@garmin.com>
Subject random HDFS exceptions
Date Tue, 16 Mar 2010 17:51:43 GMT
I am getting exceptions when trying to copy a file to HDFS. My steps are as follows -

bin/hadoop fs -rm /input/*
bin/hadoop fs -put 2010-03-15.gar /input/2010-03-15.gar

Following is the error while doing the put. I am running these steps once a day but I don't
see these errors on every run. I am running a pseudo-cluster on a commodity server. Can anyone
point me to why this may be happening and if there is any fix or workaround?


10/03/16 01:27:51 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException:
Not replicated yet:/input/2010-03-15.gar
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1253)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

        at org.apache.hadoop.ipc.Client.call(Client.java:739)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        at $Proxy0.addBlock(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at $Proxy0.addBlock(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)

10/03/16 01:27:51 WARN hdfs.DFSClient: NotReplicatedYetException sleeping /input/2010-03-15.gar
retries left 4
10/03/16 01:27:51 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException:
Not replicated yet:/input/2010-03-15.gar
{Above stack trace}

10/03/16 01:27:51 WARN hdfs.DFSClient: NotReplicatedYetException sleeping /input/2010-03-15.gar
retries left 3
10/03/16 01:27:52 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException:
Not replicated yet:/input/2010-03-15.gar
{Above stack trace}

10/03/16 01:27:52 WARN hdfs.DFSClient: NotReplicatedYetException sleeping /input/2010-03-15.gar
retries left 2
10/03/16 01:27:54 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException:
Not replicated yet:/input/2010-03-15.gar
{Above stack trace}

10/03/16 01:27:54 WARN hdfs.DFSClient: NotReplicatedYetException sleeping /input/2010-03-15.gar
retries left 1
10/03/16 01:27:57 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException:
Not replicated yet:/input/2010-03-15.gar
{Above stack trace}

10/03/16 01:27:57 INFO hdfs.DFSClient: Waiting for replication for 6 seconds
10/03/16 01:27:57 WARN hdfs.DFSClient: NotReplicatedYetException sleeping /input/2010-03-15.gar
retries left 0
10/03/16 01:28:03 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/input/2010-03-15.gar
{Above stack trace}

10/03/16 01:28:03 WARN hdfs.DFSClient: Error Recovery for block blk_-711466278859187497_1379
bad datanode[0] nodes == null
10/03/16 01:28:03 WARN hdfs.DFSClient: Could not get block locations. Source file "/input/2010-03-15.gar"
- Aborting...
put: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/input/2010-03-15.gar


--------------------------------------------------------------------------------

This e-mail and any attachments may contain confidential material for the sole use of the
intended recipient. If you are not the intended recipient, please be aware that any disclosure,
copying, distribution or use of this e-mail or any attachment is prohibited. If you have received
this e-mail in error, please contact the sender and delete all copies.



Thank you for your cooperation.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message