mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: where i can set -Dmapred.map.tasks=X
Date Mon, 03 Jan 2011 23:30:18 GMT
Could it be because of SequenceFileFromDirectory is not an AbstractJob?

On Mon, Jan 3, 2011 at 3:21 PM, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:

> I printed out arguments that it supplies to hadoop program driver:
>
> [seqdirectory, -Dfs.default.name=file:///, -Dmapred.job.tracker=local, -c,
> UTF-8, -o, /home/dmitriy/projects/testcollections/reuters-seqfiles, -i,
> /home/dmitriy/projects/testcollections/reuters-extracted/]
>
>
> So it seems to be doing the right thing with the ordering now but it still
> doesn't work for some reason with this particular command line.
>
> -Dmitriy
>
>
> On Mon, Jan 3, 2011 at 3:17 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>wrote:
>
>> Jeff,
>> now it stopped complaining about first -D but started doing so about the
>> second one.
>>
>>
>> bin/mahout seqdirectory -Dmapred.job.tracker=local -Dfs.default.name=file:///
>> -c UTF-8 -i /home/dmitriy/projects/testcollections/reuters-extracted/ -o
>> /home/dmitriy/projects/testcollections/reuters-seqfiles
>> Running on hadoop, using HADOOP_HOME=/home/dmitriy/tools/hadoop
>> No HADOOP_CONF_DIR set, using /home/dmitriy/tools/hadoop/conf
>> 11/01/03 15:16:13 ERROR text.SequenceFilesFromDirectory: Exception
>> org.apache.commons.cli2.OptionException: Unexpected -Dfs.default.name=file:///
>> while processing Options
>>
>>         at
>> org.apache.commons.cli2.commandline.Parser.parse(Parser.java:99)
>>         at
>> org.apache.mahout.text.SequenceFilesFromDirectory.main(SequenceFilesFromDirectory.java:201)
>>
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>>         at
>> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>>         at
>> org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:183)
>>
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>>
>>
>> On Mon, Jan 3, 2011 at 1:04 PM, Jeff Eastman <jdog@windwardsolutions.com>wrote:
>>
>>> Yes, I committed a small patch on the 29th. Try a new trunk build.
>>>
>>>
>>> On 1/3/11 12:37 PM, Dmitriy Lyubimov wrote:
>>>
>>>> Hi Jeff,
>>>>
>>>> so did you get around to fixing this? i am having this little bugger all
>>>> over the place, including book examples that don't work directly if i
>>>> have
>>>> hadoop setup on my machine such as in the following:
>>>>
>>>> bin/mahout seqdirectory -Dmapred.job.tracker=local -Dfs.default.name
>>>> =file:///
>>>> -c UTF-8 -i /home/dmitriy/projects/testcollections/reuters-extracted/ -o
>>>> /home/dmitriy/projects/testcollections/reuters-seqfiles
>>>> Running on hadoop, using HADOOP_HOME=/home/dmitriy/tools/hadoop
>>>> No HADOOP_CONF_DIR set, using /home/dmitriy/tools/hadoop/conf
>>>> 11/01/03 12:32:06 ERROR text.SequenceFilesFromDirectory: Exception
>>>> org.apache.commons.cli2.OptionException: Unexpected
>>>> -Dmapred.job.tracker=local while processing Options
>>>>         at
>>>> org.apache.commons.cli2.commandline.Parser.parse(Parser.java:99)
>>>>         at
>>>>
>>>> org.apache.mahout.text.SequenceFilesFromDirectory.main(SequenceFilesFromDirectory.java:187)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>         at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>         at
>>>>
>>>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>>>>         at
>>>> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>>>>         at
>>>> org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:182)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>         at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>>>>
>>>>
>>>> Thanks.
>>>> -Dmitriy
>>>>
>>>> On Wed, Dec 29, 2010 at 11:50 AM, Jeff Eastman<jeastman@narus.com>
>>>>  wrote:
>>>>
>>>>  The patch to MahoutDriver involves the code in the for loop at lines
>>>>> 203-216. If the arg.startsWith("-D") then the arg needs to be added to
>>>>> argsList at position 1, else at the end. I will commit a patch for this
>>>>> tonight as I have not got my Narus CLA signed yet.
>>>>>
>>>>> -----Original Message-----
>>>>> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
>>>>> Sent: Wednesday, December 29, 2010 11:46 AM
>>>>> To: user@mahout.apache.org
>>>>> Cc: dev@mahout.apache.org
>>>>> Subject: Re: where i can set -Dmapred.map.tasks=X
>>>>>
>>>>> ok, thank you, Jeff. Good to know. I actually expected to rely on this
>>>>> for
>>>>> a
>>>>> wide range of issues (most common being task jvm parameters override).
>>>>>
>>>>>
>>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message