hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: Command-line jobConf options in 0.18.3
Date Thu, 04 Jun 2009 21:10:53 GMT
Tom White wrote:
> Actually, the space is needed, to be interpreted as a Hadoop option by
> ToolRunner. Without the space it sets a Java system property, which
> Hadoop will not automatically pick up.

I don't think space is required. Something like 
-Dfs.default.name=host:port works. I don't see ToolRunner setting any 
java properties.

> Ian, try putting the options after the classname and see if that
> helps. Otherwise, it would be useful to see a snippet of the program
> code.

right. the options that go to the class should appear after the class name.

Note that it is not necessary to use ToolRunner (which I don't find very 
  convenient in many cases). You can use GenericOptionsParser directly. 
an example : https://issues.apache.org/jira/browse/HADOOP-5961

Raghu.

> Thanks,
> Tom
> 
> On Thu, Jun 4, 2009 at 8:23 PM, Vasyl Keretsman <vasiliy@gmail.com> wrote:
>> Perhaps, there should not be the "space" between -D and your option ?
>>
>> -Dprise.collopts=....
>>
>> Vasyl
>>
>>
>>
>> 2009/6/4 Ian Soboroff <ian.soboroff@nist.gov>:
>>> bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar gov.nist.nlpir.prise.mapred.MapReduceIndexer
input output
>>>
>>> The 'prise.collopts' option doesn't appear in the JobConf.
>>>
>>> Ian
>>>
>>> Aaron Kimball <aaron@cloudera.com> writes:
>>>
>>>> Can you give an example of the exact arguments you're sending on the command
>>>> line?
>>>> - Aaron
>>>>
>>>> On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff <ian.soboroff@nist.gov>
wrote:
>>>>
>>>>     If after I call getConf to get the conf object, I manually add the key/
>>>>     value pair, it's there when I need it.  So it feels like ToolRunner isn't
>>>>     parsing my args for some reason.
>>>>
>>>>     Ian
>>>>
>>>>     On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote:
>>>>
>>>>         Yes, and I get the JobConf via 'JobConf job = new JobConf(conf,
>>>>         the.class)'.  The conf is the Configuration object that comes from
>>>>         getConf.  Pretty much copied from the WordCount example (which this
>>>>         program used to be a long while back...)
>>>>
>>>>         thanks,
>>>>         Ian
>>>>
>>>>         On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote:
>>>>
>>>>             Are you running your program via ToolRunner.run()? How do you
>>>>             instantiate the JobConf object?
>>>>             - Aaron
>>>>
>>>>             On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff <
>>>>             ian.soboroff@nist.gov> wrote:
>>>>             I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long
>>>>             story), and I'm finding that when I run a job and try to pass
>>>>             options with -D on the command line, that the option values aren't
>>>>             showing up in my JobConf.  I logged all the key/value pairs in
the
>>>>             JobConf, and the option I passed through with -D isn't there.
>>>>
>>>>             This worked in 0.19.1... did something change with command-line
>>>>             options from 18 to 19?
>>>>
>>>>             Thanks,
>>>>             Ian


Mime
View raw message