hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amogh Vasekar <am...@yahoo-inc.com>
Subject Re: only one reduce task?
Date Fri, 04 Dec 2009 06:39:02 GMT
Hi,
If you want to access certain jobconf parameters in your streaming script, streaming provides
this by setting localized jobconf parameters as system environment variables, with the "."
in parameters replaced by "_" .
To set jobconf parameters for streaming jobs, you can use -D <param.name>=<value>

Thanks,
Amogh

On 12/4/09 6:06 AM, "Mike Kendall" <mkendall@justin.tv> wrote:

yup, only one task...

i should have mentioned that i'm using hadoop streaming.  do i have
access to jobconf* if i write my tasks in python?

On Thu, Dec 3, 2009 at 4:32 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> Mike,
>
> Do you mean you only have one reducer task for a Job ?
>
> You can increase the number of reducer task for one Job by setting
>
> JobConf.setNumReduceTasks(n)
>
>
> Jeff Zhang
>
>
> On Thu, Dec 3, 2009 at 2:58 PM, Mike Kendall <mkendall@justin.tv> wrote:
>
>> i can't seem to get my cluster to run more than one reduce task...  my
>> mapred-site.xml looks like this:
>>
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>> <!-- Put site-specific property overrides in this file. -->
>> <configuration>
>> <property>
>>  <name>mapred.job.tracker</name>
>>  <value>master:9001</value>
>> </property>
>> <property>
>>  <name>mapred.tasktracker.map.tasks.maximum</name>
>>  <value>5</value>
>> </property>
>>
>> <property>
>>  <name>mapred.tasktracker.reduce.tasks.maximum</name>
>>  <value>5</value>
>> </property>
>> <property>
>>  <name>mapred.map.tasks</name>
>>  <value>40</value>
>> </property>
>> <property>
>>  <name>mapred.reduce.tasks</name>
>>  <value>8</value>
>> </property>
>> <property>
>>  <name>mapred.jobtracker.taskScheduler</name>
>>  <value>org.apache.hadoop.mapred.FairScheduler</value>
>> </property>
>> <property>
>>  <name>mapred.fairscheduler.allocation.file</name>
>>  <value>/usr/local/hadoop/conf/fairshare-pools.xml</value>
>> </property>
>> </configuration>
>>
>> any ideas?  thanks.
>>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message