incubator-oozie-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Virag Kothari <vi...@yahoo-inc.com>
Subject Re: passing an arbitrary property to a job with map-reduce
Date Wed, 25 Jan 2012 19:18:36 GMT
Hey Neil,

If you using the old API, you have to override the configure() of Mapper.

E.g

import org.apache.hadoop.mapred.Mapper;

public class SampleMapper implements Mapper<LongWritable, Text,
LongWritable, Text> {
    
    public void configure(JobConf jobConf) {
                  System.out.println(jobConf.get("foo")); //should work
    }
}

Thanks,
Virag


On 1/25/12 11:09 AM, "Mohammad Islam" <mislam77@yahoo.com> wrote:

> One of our dev(API) has successfully  tested this case with old API.
> I would ask Virag to share his findings/usage preferably as code snippet.
> 
> Regards,
> Mohammad
> 
> 
> ----- Original Message -----
> From: Alejandro Abdelnur <tucu@cloudera.com>
> To: oozie-users@incubator.apache.org
> Cc: 
> Sent: Wednesday, January 25, 2012 10:45 AM
> Subject: Re: passing an arbitrary property to a job with map-reduce
> 
> Neil,
> 
> It looks you are using the new MR API for your mapper class.
> 
> I'd suggest start from the Oozie MR example working and modified it to
> verify that property is propagated.
> 
> Thanks.
> 
> Alejadnro
> 
> On Wed, Jan 25, 2012 at 10:12 AM, Neil Yalowitz <neilyalowitz@gmail.com>
> wrote:
>> Forgot to mention: I'm testing with Oozie v3.0.0 (but our production system
>> is actually v3.0.2).
>> 
>> Thanks,
>> Neil
>> 
>> On Wed, Jan 25, 2012 at 12:55 PM, Neil Yalowitz
>> <neilyalowitz@gmail.com>wrote:
>> 
>>> Yes, see these Pastebin links:
>>> 
>>> 
>>> workflow.xml
>>> http://pastebin.com/MqXwecjQ
>>> 
>>> ACMEMapper.java
>>> http://pastebin.com/Y9rFcU97
>>> 
>>> 
>>> Thanks!
>>> Neil
>>> 
>>> On Wed, Jan 25, 2012 at 12:23 PM, Alejandro Abdelnur
>>> <tucu@cloudera.com>wrote:
>>> 
>>>> Neil,
>>>> 
>>>> You may be missing something. This should work.
>>>> 
>>>> would you mind sharing a minimal WF.xml and MR java that shows your
>>>> problem?
>>>> 
>>>> Thanks.
>>>> 
>>>> Alejandro
>>>> 
>>>> On Wed, Jan 25, 2012 at 9:03 AM, Neil Yalowitz <neilyalowitz@gmail.com>
>>>> wrote:
>>>>> Alejandro,
>>>>> 
>>>>> Yes, we were using the new MR API, so I modified the workflow.xml to
>>>> remove
>>>>> mapred.mapper.new-api and re-tested.  I still cannot pass a property
to
>>>> my
>>>>> MR job.
>>>>> 
>>>>> Is passing a non mapred.* property to a <map-reduce> job supported
with
>>>>> Oozie?  Or must I use the <java> job and parse the property in
my
>>>> main()?
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> Neil
>>>>> 
>>>>> On Tue, Jan 24, 2012 at 6:00 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>> wrote:
>>>>> 
>>>>>> Neil,
>>>>>> 
>>>>>> Are you trying to use the new MR API?
>>>>>> 
>>>>>> If so please look at
>>>>>> 
>>>>>> https://github.com/yahoo/oozie/wiki/Oozie-WF-use-cases
>>>>>> 
>>>>>> The "How to run Map-reduce job written using new Hadoop API?" section.
>>>>>> 
>>>>>> Thanks.
>>>>>> 
>>>>>> Alejandro
>>>>>> 
>>>>>> On Tue, Jan 24, 2012 at 12:14 PM, Neil Yalowitz <
>>>> neilyalowitz@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> I'm attempting to pass a property to an MR job with an oozie
workflow
>>>>>>> structured with a <map-reduce> tag.  This was trivial
with a <java>
>>>> tag,
>>>>>> as
>>>>>>> I would call my own main(String[] args) and instantiate my own
>>>> Driver and
>>>>>>> then ToolRunner.run().  Because I was coding my own main(),
I had an
>>>>>>> opportunity to parse/set properties easily:
>>>>>>> 
>>>>>>> workflow.xml
>>>>>>> <java>
>>>>>>>  <main-class>com.acme.MyMain</main-class>
>>>>>>>  <configuration>
>>>>>>>    ...
>>>>>>>  </configuration>
>>>>>>>  <arg>foovalue</arg>
>>>>>>>  <arg>barvalue</arg>
>>>>>>> </java>
>>>>>>> 
>>>>>>> ...and capture those arguments:
>>>>>>> 
>>>>>>> MyMain.java
>>>>>>> public static void main(String[] args) throws Exception {
>>>>>>>  ...
>>>>>>>  String foo = args[0];
>>>>>>>  String bar = args[1];
>>>>>>>  String[] argsAll = {foo, bar};
>>>>>>>  Configuration conf = new Configuration();
>>>>>>>  ToolRunner.run(conf, new MyDriver(), argsAll);
>>>>>>> }
>>>>>>> 
>>>>>>> This worked fine, but I can't figure out how to capture those
>>>> properties
>>>>>>> using a <map-reduce> tag.  I attempt something like this:
>>>>>>> 
>>>>>>> workflow.xml
>>>>>>> <map-reduce>
>>>>>>>  ...
>>>>>>>  <configuration>
>>>>>>>    <property>
>>>>>>>      <name>foo</name>
>>>>>>>      <value>foovalue</value>
>>>>>>>    </property>
>>>>>>>  </configuration>
>>>>>>> </map-reduce>
>>>>>>> 
>>>>>>> However, the foo property is not set in the created
>>>>>>> org.apache.hadoop.conf.Configuration object:
>>>>>>> 
>>>>>>> MyMapper.java
>>>>>>> ...
>>>>>>> @Override
>>>>>>> protected void setup(Context context)... {
>>>>>>>  context.getConfiguration().get("foo"); //null!
>>>>>>> }
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Is there a way to pass an arbitrary property (ie-- not a mapred.*
>>>>>> property)
>>>>>>> to a job with an Oozie workflow using a <map-reduce> tag?
 Perhaps I
>>>> am
>>>>>>> missing something obvious...
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks!
>>>>>>> 
>>>>>>> 
>>>>>>> Neil Yalowitz
>>>>>> 
>>>> 
>>> 
>>> 
> 


Mime
View raw message