Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=iSOlE3H1486OTRXdxzGYM9I/evFl/0MtxJqiRwuYFoeMqO/ZNnn8aV4ydZspf5L3
Message-ID: <4A28385D.6090301@yahoo-inc.com>
Date: Thu, 04 Jun 2009 14:10:53 -0700
From: Raghu Angadi <rangadi@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: core-user@hadoop.apache.org
Subject: Re: Command-line jobConf options in 0.18.3
References: <0EA60242-483B-4CE6-B4FF-2C5391F028DA@nist.gov>
	 <d6d7c4410906031609p46ec905rc1cb0beb86ca7aa7@mail.gmail.com>
	 <E8916F57-8982-4702-B31E-51660838E849@nist.gov>
	 <DD50EBD1-146E-4255-8EC7-88B89F70B493@nist.gov>
	 <d6d7c4410906040111r61ff0310p6a7cc032b06fb0ca@mail.gmail.com>
	 <9cfbpp412bi.fsf@rogue.ncsl.nist.gov>
	 <888c90ec0906041223t5cb8504fh3621ff91039bf867@mail.gmail.com>
 <ac79ea400906041251xe84e11di9b92779c2ebf0c57@mail.gmail.com>
In-Reply-To: <ac79ea400906041251xe84e11di9b92779c2ebf0c57@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Tom White wrote:
> Actually, the space is needed, to be interpreted as a Hadoop option by
> ToolRunner. Without the space it sets a Java system property, which
> Hadoop will not automatically pick up.

I don't think space is required. Something like 
-Dfs.default.name=host:port works. I don't see ToolRunner setting any 
java properties.

> Ian, try putting the options after the classname and see if that
> helps. Otherwise, it would be useful to see a snippet of the program
> code.

right. the options that go to the class should appear after the class name.

Note that it is not necessary to use ToolRunner (which I don't find very 
  convenient in many cases). You can use GenericOptionsParser directly. 
an example : https://issues.apache.org/jira/browse/HADOOP-5961

Raghu.

> Thanks,
> Tom
> 
> On Thu, Jun 4, 2009 at 8:23 PM, Vasyl Keretsman <vasiliy@gmail.com> wrote:
>> Perhaps, there should not be the "space" between -D and your option ?
>>
>> -Dprise.collopts=....
>>
>> Vasyl
>>
>>
>>
>> 2009/6/4 Ian Soboroff <ian.soboroff@nist.gov>:
>>> bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar gov.nist.nlpir.prise.mapred.MapReduceIndexer input output
>>>
>>> The 'prise.collopts' option doesn't appear in the JobConf.
>>>
>>> Ian
>>>
>>> Aaron Kimball <aaron@cloudera.com> writes:
>>>
>>>> Can you give an example of the exact arguments you're sending on the command
>>>> line?
>>>> - Aaron
>>>>
>>>> On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff <ian.soboroff@nist.gov> wrote:
>>>>
>>>>     If after I call getConf to get the conf object, I manually add the key/
>>>>     value pair, it's there when I need it.  So it feels like ToolRunner isn't
>>>>     parsing my args for some reason.
>>>>
>>>>     Ian
>>>>
>>>>     On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote:
>>>>
>>>>         Yes, and I get the JobConf via 'JobConf job = new JobConf(conf,
>>>>         the.class)'.  The conf is the Configuration object that comes from
>>>>         getConf.  Pretty much copied from the WordCount example (which this
>>>>         program used to be a long while back...)
>>>>
>>>>         thanks,
>>>>         Ian
>>>>
>>>>         On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote:
>>>>
>>>>             Are you running your program via ToolRunner.run()? How do you
>>>>             instantiate the JobConf object?
>>>>             - Aaron
>>>>
>>>>             On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff <
>>>>             ian.soboroff@nist.gov> wrote:
>>>>             I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long
>>>>             story), and I'm finding that when I run a job and try to pass
>>>>             options with -D on the command line, that the option values aren't
>>>>             showing up in my JobConf.  I logged all the key/value pairs in the
>>>>             JobConf, and the option I passed through with -D isn't there.
>>>>
>>>>             This worked in 0.19.1... did something change with command-line
>>>>             options from 18 to 19?
>>>>
>>>>             Thanks,
>>>>             Ian