incubator-oozie-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject Re: passing an arbitrary property to a job with map-reduce
Date Tue, 24 Jan 2012 23:00:08 GMT
Neil,

Are you trying to use the new MR API?

If so please look at

https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases

The "How to run Map-reduce job written using new Hadoop API?" section.

Thanks.

Alejandro

On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <neilyalowitz@gmail.com> wrote:
>
> Hi all,
>
> I'm attempting to pass a property to an MR job with an oozie workflow
> structured with a <map-reduce> tag.  This was trivial with a <java> tag,
as
> I would call my own main(String[] args) and instantiate my own Driver and
> then ToolRunner.run().  Because I was coding my own main(), I had an
> opportunity to parse/set properties easily:
>
> workflow.xml
> <java>
>  <main-class>com.acme.MyMain</main-class>
>  <configuration>
>    ...
>  </configuration>
>  <arg>foovalue</arg>
>  <arg>barvalue</arg>
> </java>
>
> ...and capture those arguments:
>
> MyMain.java
> public static void main(String[] args) throws Exception {
>  ...
>  String foo = args[0];
>  String bar = args[1];
>  String[] argsAll = {foo, bar};
>  Configuration conf = new Configuration();
>  ToolRunner.run(conf, new MyDriver(), argsAll);
> }
>
> This worked fine, but I can't figure out how to capture those properties
> using a <map-reduce> tag.  I attempt something like this:
>
> workflow.xml
> <map-reduce>
>  ...
>  <configuration>
>    <property>
>      <name>foo</name>
>      <value>foovalue</value>
>    </property>
>  </configuration>
> </map-reduce>
>
> However, the foo property is not set in the created
> org.apache.hadoop.conf.Configuration object:
>
> MyMapper.java
> ...
> @Override
> protected void setup(Context context)... {
>  context.getConfiguration().get("foo"); //null!
> }
>
>
>
> Is there a way to pass an arbitrary property (ie-- not a mapred.* property)
> to a job with an Oozie workflow using a <map-reduce> tag?  Perhaps I am
> missing something obvious...
>
>
>
> Thanks!
>
>
> Neil Yalowitz

Mime
View raw message