hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abees Muhammad <abees...@gmail.com>
Subject Re: Execute a Map/Reduce Job Jar from Another Java Program.
Date Fri, 03 Feb 2012 05:31:24 GMT
Hi Evans,

Thanks for your reply. I have a mapreduce job jar file lets call it as
test.jar. I am executing this jar file as hadoop jar test.jar inputpath
outPath, and it is executed succesfully. Now i want to execute this job for
a batch of files(a batch of 20 files), for this purpose i have created
another java application,this application moves a batch of files from one
location of hdfs to another location in hdfs. After that this application
needs to execute the m/R job for this batch. we will invoke the second
application(which will execute the M/R Job) from as control m job.But i
dont know how to create the second java application which will invoke the
M/R job. The code snippet i used for testing the jar which calls the M/R
job is

        List arguments = new ArrayList();
        arguments.add("test.jar");
        arguments.add("inputPath");
        arguments.add(outputPath);
        RunJar.main((String[])arguments.toArray(new String[0]));

i executed this jar as java -jar M/RJobInvokeApp.jar but i got error as

java.lang.IllegalArgumentException: Wrong FS: hdfs://ip
address:54310/tmp/test-out, expected: file:///
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:410)
    at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:56)
    at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:379)
    at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:748)
    at
com.amd.kdf.protobuf.SequentialFileDriver.main(SequentialFileDriver.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
    at com.amd.wrapper.main.ParserWrapper.main(ParserWrapper.java:31)






Thanks,
Abees

On 2 February 2012 23:02, Robert Evans <evans@yahoo-inc.com> wrote:

>  What happens?  Is there an exception, does nothing happen?  I am
> curious.  Also how did you launch your other job that is trying to run this
> one.  The hadoop script sets up a lot of environment variables classpath
> etc to make hadoop work properly, and some of that may not be set up
> correctly to make RunJar work.
>
> --Bobby Evans
>
>
> On 2/2/12 9:36 AM, "Harsh J" <harsh@cloudera.com> wrote:
>
> Moving to common-user. Common-dev is for project development
> discussions, not user help.
>
> Could you elaborate on how you used RunJar? What arguments did you
> provide, and is the target jar a runnable one or a regular jar? What
> error did you get?
>
> On Thu, Feb 2, 2012 at 8:44 PM, abees muhammad <abees.pm@gmail.com> wrote:
> >
> > Hi,
> >
> > I am a newbie to Hadoop Development. I have a Map/Reduce job jar file, i
> > want to execute this jar file programmatically from another java
> program. I
> > used the following code to execute it.
> >
> > RunJar.main(String[] args). But The jar file is not executed.
> >
> > Can you please give me  a work around for this issue.
> > --
> > View this message in context:
> http://old.nabble.com/Execute-a-Map-Reduce-Job-Jar-from-Another-Java-Program.-tp33250801p33250801.html
> > Sent from the Hadoop core-dev mailing list archive at Nabble.com.
> >
>
>
>
> --
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message