hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amareshwari Sriramadasu <amar...@yahoo-inc.com>
Subject Re: Running mapred job from remote machine to a pseudo-distributed hadoop
Date Mon, 04 Aug 2008 04:54:48 GMT
Arv Mistry wrote:
> I'll try again, can anyone tell me should it be possible to run hadoop
> in a pseudo-distributed mode (i.e. everything on one machine) and then
> submit a mapred job using the ToolRunner from another machine on that
> hadoop configuration?
>
> Cheers Arv
>  
>   
Yes. It is possible to do. You can start hadoop cluster on single node.
Documentation available at 
http://hadoop.apache.org/core/docs/current/quickstart.html#PseudoDistributed
Once the cluster is up, you can submit jobs from any client, but the 
client configuration should be aware of Namenode and JobTracker nodes. 
You can use the generic options *-fs* and *-jt* on commandline for the same.

Thanks
Amareshwari
>
> -----Original Message-----
> From: Arv Mistry [mailto:arv@kindsight.net] 
> Sent: Thursday, July 31, 2008 2:32 PM
> To: core-user@hadoop.apache.org
> Subject: Running mapred job from remote machine to a pseudo-distributed
> hadoop
>
>  
> I have hadoop setup in a pseudo-distributed mode i.e. everything on one
> machine, And I'm trying to submit a hadoop mapred job from another
> machine to that hadoop setup.
>
> At the point that I run the mapred job I get the following error. Any
> ideas as to what I'm doing wrong?
> Is this possible in a pseudo-distributed mode?
>
> Cheers Arv
>
>  INFO   | jvm 1    | 2008/07/31 14:01:00 | 2008-07-31 14:01:00,547 ERROR
> [HadoopJobTool] java.io.IOException:
> /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such
> file or directory
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:175)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> java.lang.reflect.Method.invoke(Method.java:597)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |
> INFO   | jvm 1    | 2008/07/31 14:01:00 |
> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> /tmp/hadoop-root/mapred/system/job_200807310809_0006/job.xml: No such
> file or directory
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:149)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1155)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1136)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:175)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> java.lang.reflect.Method.invoke(Method.java:597)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.ipc.Client.call(Client.java:557)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> $Proxy5.submitJob(Unknown Source)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> java.lang.reflect.Method.invoke(Method.java:597)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo
> cationHandler.java:82)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation
> Handler.java:59)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> $Proxy5.submitJob(Unknown Source)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:758)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> com.rialto.profiler.profiler.clickstream.hadoop.HadoopJobTool.run(Hadoop
> JobTool.java:129)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> com.rialto.profiler.profiler.clickstream.hadoop.HadoopJobTool.launchJob(
> HadoopJobTool.java:142)
> INFO   | jvm 1    | 2008/07/31 14:01:00 |       at
> com.rialto.profiler.profiler.clickstream.RawStreamGenerator.run(RawStrea
> mGenerator.java:138)
>   


Mime
View raw message