hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Istabrak Abdul-Fatah <ifa...@gmail.com>
Subject Re: Passing Args to Mappers and Reducers
Date Tue, 06 Oct 2015 15:19:25 GMT
Hi Naga,
Thx for the follow up.
Still not working.
Here are the invocation errors and the source code.

[image: Inline image 1]


public class AvgSix extends Configured implements Tool{

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("test", System.getProperty("ista"));  // <=== This is the line
where it is failing
int res = ToolRunner.run(conf, new AvgSix(), args);
System.exit(res);
}

[yarn@caotclc04881 ~]$ yarn jar /opt/yarn/my_examples/AvgSix.jar
-Dista="667788" /user/yarn/input/samplePMfile.v31.csv output Exception in
thread "main" java.lang.IllegalArgumentException: The value of property
test must not be null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136) [yarn@caotclc04881 ~]$
yarn jar /opt/yarn/my_examples/AvgSix.jar AvgSix -Dista="667788"
/user/yarn/input/samplePMfile.v31.csv output Exception in thread "main"
java.lang.IllegalArgumentException: The value of property test must not be
null at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at
org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at
AvgSix.main(AvgSix.java:32) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136)


On Tue, Oct 6, 2015 at 10:49 AM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:

> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> I think in your case it should be "*yarn  jar
> /opt/yarn/my_examples/AvgSix.jar <classname if req> -Dista="666666"
> /user/yarn/input/samplefile.csv output*"
>
> + Naga
> ------------------------------
> *From:* Naganarasimha G R (Naga)
> *Sent:* Tuesday, October 06, 2015 20:16
> *To:* user@hadoop.apache.org
> *Subject:* RE: Passing Args to Mappers and Reducers
>
> Hi Ista,
>
> In general we need to give after specifying the class name like
> *./yarn jar
> ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar
> sleep -Dmapreduce.job.queuename=default -m 10 -r 1 -mt 5000*
>
> i think in your case it should be
> ------------------------------
> *From:* Istabrak Abdul-Fatah [ifatah@gmail.com]
> *Sent:* Tuesday, October 06, 2015 19:46
> *To:* user@hadoop.apache.org
> *Subject:* Re: Passing Args to Mappers and Reducers
>
> Hi Shahab,
> Thx for the quick reply.
> Yes, I am using the tool interface and can run the job successfully.
> I need to pass some extra args dynamically as part of the invocation.
>
> I have tried adding a space after the -D option and it did not work either
> (see below).
>
> ~]$ yarn -D ista="666666" jar /opt/yarn/my_examples/AvgSix.jar
> /user/yarn/input/sample.csv output
> Error: Could not find or load main class ista=666666
>
> It does not see to recognize the -D option.
>
> BR
>
> Ista
>
> On Tue, Oct 6, 2015 at 9:26 AM, Shahab Yunus <shahab.yunus@gmail.com>
> wrote:
>
>> Are you properly implementing the Tool interface?
>>
>> https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/
>>
>> Also, there needs to be space between -D and the param name.
>>
>> Regards,
>> Shahab
>>
>> On Tue, Oct 6, 2015 at 9:22 AM, Istabrak Abdul-Fatah <ifatah@gmail.com>
>> wrote:
>>
>>> Greetings to all,
>>> Is it possible to pass args to Mapper and Reducers via the command line
>>> args.
>>> I need to pass some args upon the MapRed job invocation so that I can
>>> pass these args via the Context object to the Mapper and Reducer code.
>>> I am currently running Hadoop2.7 and tried to pass some args using the
>>> -D VM args but it did not succeed.
>>> Here are some failed code snippets:
>>>
>>> ~]$ yarn -Dista="666666" jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn  jar -Dista="666666" /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output
>>>
>>> ~]$ yarn  jar /opt/yarn/my_examples/AvgSix.jar
>>> /user/yarn/input/samplefile.csv output  -Dista="666666"
>>>
>>> Please advise.
>>>
>>>
>>> Thx and BR
>>>
>>> Ista
>>>
>>
>>
>

Mime
View raw message