bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Fryer <dfryer1...@gmail.com>
Subject Re: Hadoop Single-Node Problems
Date Mon, 28 Jul 2014 17:50:24 GMT
Hey,
I actually moved to using puppet to deploy at the end of last week. I
managed to get everything up and working this morning, with some help from
Jay. It ended up being permissions issues, in addition to not knowing that
the commands need to be run as the user hdfs, as opposed to root.

Would it be possible for me to have wiki-editing permissions in order to
document this?

-David Fryer


On Mon, Jul 28, 2014 at 1:11 PM, Konstantin Boudnik <cos@apache.org> wrote:

> IIRC you were installing the cluster manually, as in no-Puppet. As the
> result you don't have HDFS structured properly. The result - you have no
> /user/root created (and that's what will be used when you do 'sudo hadoop
> something').
>
> You can fix it easily by locating and running init-hdfs.sh script under
> hadoop
> installation directory.
>
> Cos
>
> On Mon, Jul 28, 2014 at 09:16AM, David Fryer wrote:
> > Hi Bigtop,
> >
> > When I try to run an example job on a single-node, I get the following
> > exceptions:
> > [hadoopuser@master bin]$ sudo hadoop jar
> > /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 1000
> > Number of Maps  = 10
> > Samples per Map = 1000
> > 14/07/28 07:45:33 WARN mapred.JobConf: The variable mapred.child.ulimit
> is
> > no longer used.
> > org.apache.hadoop.security.AccessControlException: Permission denied:
> > user=root, access=WRITE, inode="/":hdfs:hadoop:drwxr-xr-x
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:221)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:201)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:146)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4546)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4516)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:2936)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:2900)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2882)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:659)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
> > at
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40760)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> > at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at
> >
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
> > at
> >
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
> > at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2144)
> > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2113)
> > at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:540)
> > at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1881)
> > at
> >
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:282)
> > at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > Caused by:
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> > Permission denied: user=root, access=WRITE,
> inode="/":hdfs:hadoop:drwxr-xr-x
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:221)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:201)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:146)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4546)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:4516)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:2936)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:2900)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:2882)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:659)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
> > at
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40760)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:1240)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> > at com.sun.proxy.$Proxy9.mkdirs(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> > at com.sun.proxy.$Proxy9.mkdirs(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:446)
> > at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2142)
> > ... 19 more
> >
> > Also, I get an exception when I try to initialize hdfs:
> > [hadoopuser@master bin]$ sudo /usr/lib/hadoop/libexec/init-hdfs.sh
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /tmp'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 1777 /tmp'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /var'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /var/log'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 1775 /var/log'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown yarn:mapred
> /var/log'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /tmp/hadoop-yarn'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown -R mapred:mapred
> > /tmp/hadoop-yarn'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 777
> > /tmp/hadoop-yarn'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir -p
> > /var/log/hadoop-yarn/apps'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 1777
> > /var/log/hadoop-yarn/apps'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown yarn:mapred
> > /var/log/hadoop-yarn/apps'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /hbase'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown hbase:hbase /hbase'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /solr'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown solr:solr /solr'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /benchmarks'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 777 /benchmarks'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /user'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod 755 /user'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown hdfs  /user'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /user/history'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown mapred:mapred
> > /user/history'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod 755 /user/history'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /user/jenkins'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 777
> /user/jenkins'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown jenkins
> /user/jenkins'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /user/hive'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 777 /user/hive'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown hive /user/hive'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /user/root'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 777 /user/root'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown root /user/root'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /user/hue'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 777 /user/hue'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown hue /user/hue'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /user/sqoop'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 777 /user/sqoop'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown sqoop /user/sqoop'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /user/oozie'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 777 /user/oozie'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown -R oozie
> /user/oozie'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir /user/oozie/share'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir
> /user/oozie/share/lib'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir
> > /user/oozie/share/lib/hive'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir
> > /user/oozie/share/lib/mapreduce-streaming'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir
> > /user/oozie/share/lib/distcp'
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir
> > /user/oozie/share/lib/pig'
> > + ls '/usr/lib/hive/lib/*.jar'
> > + ls /usr/lib/hadoop-mapreduce/hadoop-streaming-2.0.6-alpha.jar
> > /usr/lib/hadoop-mapreduce/hadoop-streaming.jar
> > + su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -put
> > /usr/lib/hadoop-mapreduce/hadoop-streaming*.jar
> > /user/oozie/share/lib/mapreduce-streaming'
> > 14/07/28 07:58:35 WARN hdfs.DFSClient: DataStreamer Exception
> > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> >
> /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.0.6-alpha.jar._COPYING_
> > could only be replicated to 0 nodes instead of minReplication (=1).
>  There
> > are 0 datanode(s) running and no node(s) are excluded in this operation.
> > at
> >
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1339)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2155)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:491)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:351)
> > at
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40744)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:1240)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> > at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> > at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:311)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1156)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1009)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
> > put: File
> >
> /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.0.6-alpha.jar._COPYING_
> > could only be replicated to 0 nodes instead of minReplication (=1).
>  There
> > are 0 datanode(s) running and no node(s) are excluded in this operation.
> > 14/07/28 07:58:35 WARN hdfs.DFSClient: DataStreamer Exception
> > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> > /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming.jar._COPYING_
> > could only be replicated to 0 nodes instead of minReplication (=1).
>  There
> > are 0 datanode(s) running and no node(s) are excluded in this operation.
> > at
> >
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1339)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2155)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:491)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:351)
> > at
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40744)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:1240)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> > at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> > at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:311)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1156)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1009)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
> > put: File
> > /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming.jar._COPYING_
> > could only be replicated to 0 nodes instead of minReplication (=1).
>  There
> > are 0 datanode(s) running and no node(s) are excluded in this operation.
> > 14/07/28 07:58:35 ERROR hdfs.DFSClient: Failed to close file
> >
> /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.0.6-alpha.jar._COPYING_
> > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> >
> /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.0.6-alpha.jar._COPYING_
> > could only be replicated to 0 nodes instead of minReplication (=1).
>  There
> > are 0 datanode(s) running and no node(s) are excluded in this operation.
> > at
> >
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1339)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2155)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:491)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:351)
> > at
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40744)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:1240)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> > at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> > at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:311)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1156)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1009)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
> > 14/07/28 07:58:35 ERROR hdfs.DFSClient: Failed to close file
> > /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming.jar._COPYING_
> > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> > /user/oozie/share/lib/mapreduce-streaming/hadoop-streaming.jar._COPYING_
> > could only be replicated to 0 nodes instead of minReplication (=1).
>  There
> > are 0 datanode(s) running and no node(s) are excluded in this operation.
> > at
> >
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1339)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2155)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:491)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:351)
> > at
> >
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40744)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1741)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1737)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1735)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:1240)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> > at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> > at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:311)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1156)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1009)
> > at
> >
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
> >
> > Can anyone help resolve these issues?
> >
> > Thank You,
> > David Fryer
>

Mime
View raw message