hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: Can not follow Single Node Setup example.
Date Thu, 27 Jun 2013 18:20:23 GMT
Not an issue. See, there are 2 types of modes when you say "single node
setup" : standalone(runs on your local FS) and pseudo distributed(runs on
HDFS). You are probably working on standalone setup. If you need some help
on pseudo setup you might this link helpful :
http://cloudfront.blogspot.in/2012/07/how-to-configure-hadoop.html#.UcyBE0AW38s

I have tried to explain the procedure.

Warm Regards,
Tariq
cloudfront.blogspot.com


On Thu, Jun 27, 2013 at 11:41 PM, Peng Yu <pengyu.ut@gmail.com> wrote:

> I just started learning hadoop. And I followed
> http://hadoop.apache.org/docs/r1.1.2/single_node_setup.html. Is
> DataNode mentioned in this document? Do you have a list of working
> step by step instructions so that I run hadoop without anything
> previously installed? Thanks.
>
> On Thu, Jun 27, 2013 at 1:00 PM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> > Is your DataNode running?
> >
> > Warm Regards,
> > Tariq
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Jun 27, 2013 at 11:24 PM, Peng Yu <pengyu.ut@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> Here is what I got. Is there anything wrong?
> >>
> >> ~/Downloads/hadoop-install/hadoop$ bin/hadoop fs -put conf/  /input/
> >> 13/06/27 12:53:39 WARN hdfs.DFSClient: DataStreamer Exception:
> >> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> /input/conf/capacity-scheduler.xml could only be replicated to 0
> >> nodes, instead of 1
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
> >>         at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
> >>         at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>         at java.lang.reflect.Method.invoke(Method.java:597)
> >>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> >>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> >>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> >>         at java.security.AccessController.doPrivileged(Native Method)
> >>         at javax.security.auth.Subject.doAs(Subject.java:396)
> >>         at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> >>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
> >>
> >>         at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> >>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> >>         at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
> >>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>         at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>         at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>         at java.lang.reflect.Method.invoke(Method.java:597)
> >>         at
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> >>         at
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> >>         at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
> >>
> >> 13/06/27 12:53:39 WARN hdfs.DFSClient: Error Recovery for block null
> >> bad datanode[0] nodes == null
> >> 13/06/27 12:53:39 WARN hdfs.DFSClient: Could not get block locations.
> >> Source file "/input/conf/capacity-scheduler.xml" - Aborting...
> >> put: java.io.IOException: File /input/conf/capacity-scheduler.xml
> >> could only be replicated to 0 nodes, instead of 1
> >> 13/06/27 12:53:39 ERROR hdfs.DFSClient: Failed to close file
> >> /input/conf/capacity-scheduler.xml
> >> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> /input/conf/capacity-scheduler.xml could only be replicated to 0
> >> nodes, instead of 1
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639)
> >>         at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736)
> >>         at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
> >>         at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>         at java.lang.reflect.Method.invoke(Method.java:597)
> >>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> >>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> >>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> >>         at java.security.AccessController.doPrivileged(Native Method)
> >>         at javax.security.auth.Subject.doAs(Subject.java:396)
> >>         at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> >>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
> >>
> >>         at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> >>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> >>         at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
> >>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>         at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>         at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>         at java.lang.reflect.Method.invoke(Method.java:597)
> >>         at
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> >>         at
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> >>         at com.sun.proxy.$Proxy1.addBlock(Unknown Source)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
> >>
> >> On Thu, Jun 27, 2013 at 12:40 PM, Mohammad Tariq <dontariq@gmail.com>
> >> wrote:
> >> > No. This means that you are trying to copy an entire directory instead
> >> > of a
> >> > file. Do this :
> >> > bin/hadoop fs -put conf/  /input/
> >> >
> >> > Warm Regards,
> >> > Tariq
> >> > cloudfront.blogspot.com
> >> >
> >> >
> >> > On Thu, Jun 27, 2013 at 10:37 PM, Peng Yu <pengyu.ut@gmail.com>
> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> ~/Downloads/hadoop-install/hadoop$ rm -rf ~/input/conf/
> >> >> ~/Downloads/hadoop-install/hadoop$ bin/hadoop fs -put conf input
> >> >> put: Target input/conf is a directory
> >> >>
> >> >> I get the above output. Is it the correct output? Thanks.
> >> >>
> >> >> On Wed, Jun 26, 2013 at 10:51 AM, Shahab Yunus <
> shahab.yunus@gmail.com>
> >> >> wrote:
> >> >> > It is looking for a file within your login folder
> >> >> > /user/py/input/conf
> >> >> >
> >> >> > You are running your job form
> >> >> > hadoop/bin
> >> >> > and I think the hadoop job will is looking for files in the current
> >> >> > folder.
> >> >> >
> >> >> > Regards,
> >> >> > Shahab
> >> >> >
> >> >> >
> >> >> > On Wed, Jun 26, 2013 at 11:02 AM, Peng Yu <pengyu.ut@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi,
> >> >> >>
> >> >> >> Here are what I have.
> >> >> >>
> >> >> >> ~/Downloads/hadoop-install/hadoop$ ls
> >> >> >> CHANGES.txt  README.txt  c++      hadoop-ant-1.1.2.jar
> >> >> >> hadoop-examples-1.1.2.jar     hadoop-tools-1.1.2.jar  ivy.xml
>  logs
> >> >> >> src
> >> >> >> LICENSE.txt  bin         conf     hadoop-client-1.1.2.jar
> >> >> >> hadoop-minicluster-1.1.2.jar  input                   lib
>  sbin
> >> >> >> webapps
> >> >> >> NOTICE.txt   build.xml   contrib  hadoop-core-1.1.2.jar
> >> >> >> hadoop-test-1.1.2.jar         ivy                     libexec
>  share
> >> >> >> ~/Downloads/hadoop-install/hadoop$ ls input/
> >> >> >> capacity-scheduler.xml  core-site.xml  fair-scheduler.xml
> >> >> >> hadoop-policy.xml  hdfs-site.xml  mapred-queue-acls.xml
> >> >> >> mapred-site.xml
> >> >> >>
> >> >> >> On Wed, Jun 26, 2013 at 10:00 AM, Shahab Yunus
> >> >> >> <shahab.yunus@gmail.com>
> >> >> >> wrote:
> >> >> >> > Basically whether this step worked or not:
> >> >> >> >
> >> >> >> > $ cp conf/*.xml input
> >> >> >> >
> >> >> >> > Regards,
> >> >> >> > Shahab
> >> >> >> >
> >> >> >> >
> >> >> >> > On Wed, Jun 26, 2013 at 10:58 AM, Shahab Yunus
> >> >> >> > <shahab.yunus@gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Have you verified that the 'input' folder exists
on the hdfs
> >> >> >> >> (singel
> >> >> >> >> node
> >> >> >> >> setup) that you are job needs?
> >> >> >> >>
> >> >> >> >> Regards,
> >> >> >> >> Shahab
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Wed, Jun 26, 2013 at 10:53 AM, Peng Yu <pengyu.ut@gmail.com
> >
> >> >> >> >> wrote:
> >> >> >> >>>
> >> >> >> >>> Hi,
> >> >> >> >>>
> >> >> >> >>> http://hadoop.apache.org/docs/r1.1.2/single_node_setup.html
> >> >> >> >>>
> >> >> >> >>> I followed the above instructions. But I get
the following
> >> >> >> >>> errors.
> >> >> >> >>> Does anybody know what is wrong? Thanks.
> >> >> >> >>>
> >> >> >> >>> ~/Downloads/hadoop-install/hadoop$ bin/hadoop
jar
> >> >> >> >>> hadoop-examples-*.jar grep input output 'dfs[a-z.]+'
> >> >> >> >>> Warning: $HADOOP_HOME is deprecated.
> >> >> >> >>>
> >> >> >> >>> 13/06/26 09:49:14 WARN util.NativeCodeLoader:
Unable to load
> >> >> >> >>> native-hadoop library for your platform... using
builtin-java
> >> >> >> >>> classes
> >> >> >> >>> where applicable
> >> >> >> >>> 13/06/26 09:49:14 WARN snappy.LoadSnappy: Snappy
native
> library
> >> >> >> >>> not
> >> >> >> >>> loaded
> >> >> >> >>> 13/06/26 09:49:14 INFO mapred.FileInputFormat:
Total input
> paths
> >> >> >> >>> to
> >> >> >> >>> process : 2
> >> >> >> >>> 13/06/26 09:49:14 INFO mapred.JobClient: Cleaning
up the
> staging
> >> >> >> >>> area
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> hdfs://localhost:9000/opt/local/var/hadoop/cache/mapred/staging/py/.staging/job_201306260838_0001
> >> >> >> >>> 13/06/26 09:49:14 ERROR security.UserGroupInformation:
> >> >> >> >>> PriviledgedActionException as:py cause:java.io.IOException:
> Not
> >> >> >> >>> a
> >> >> >> >>> file: hdfs://localhost:9000/user/py/input/conf
> >> >> >> >>> java.io.IOException: Not a file:
> >> >> >> >>> hdfs://localhost:9000/user/py/input/conf
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:215)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1051)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1043)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
> >> >> >> >>>         at
> >> >> >> >>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:959)
> >> >> >> >>>         at
> >> >> >> >>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
> >> >> >> >>>         at java.security.AccessController.doPrivileged(Native
> >> >> >> >>> Method)
> >> >> >> >>>         at javax.security.auth.Subject.doAs(Subject.java:396)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
> >> >> >> >>>         at
> >> >> >> >>>
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:886)
> >> >> >> >>>         at
> >> >> >> >>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323)
> >> >> >> >>>         at org.apache.hadoop.examples.Grep.run(Grep.java:69)
> >> >> >> >>>         at
> >> >> >> >>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >> >> >> >>>         at org.apache.hadoop.examples.Grep.main(Grep.java:93)
> >> >> >> >>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >> >> >>> Method)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >> >> >>>         at java.lang.reflect.Method.invoke(Method.java:597)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> >> >> >> >>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >> >> >>> Method)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >> >> >>>         at
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >> >> >>>         at java.lang.reflect.Method.invoke(Method.java:597)
> >> >> >> >>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> Regards,
> >> >> >> >>> Peng
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Regards,
> >> >> >> Peng
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Regards,
> >> >> Peng
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Peng
> >
> >
>
>
>
> --
> Regards,
> Peng
>

Mime
View raw message