hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From james warren <ja...@rockyou.com>
Subject Re: fair scheduler preemptions timeout difficulties
Date Thu, 03 Dec 2009 00:55:41 GMT
Todd from Cloudera solved this for me on their company's forum.

"What you're missing is the "mapred.fairscheduler.preemption" property in
mapred-site.xml - without this on, the preemption settings in the
allocations file are ignored... to turn it on, set that property's value to
'true'"

Thanks, Todd!

On Wed, Dec 2, 2009 at 4:26 PM, james warren <james@rockyou.com> wrote:

> Greetings, Hadoop Fans:
>
> I'm attempting to use the timeout feature of the Fair Scheduler (using
> Cloudera's most recently released distribution 0.20.1+152-1), but without
> success.  I'm using the following configs:
>
> /etc/hadoop/conf/mapred-site.xml
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <configuration>
>   <property>
>     <name>mapred.job.tracker</name>
>     <value>hadoop-master:8021</value>
>   </property>
>   <property>
>      <name>mapred.tasktracker.map.tasks.maximum</name>
>      <value>9</value>
>   </property>
>   <property>
>      <name>mapred.tasktracker.reduce.tasks.maximum</name>
>      <value>3</value>
>   </property>
>   <property>
>      <name>mapred.jobtracker.taskScheduler</name>
>      <value>org.apache.hadoop.mapred.FairScheduler</value>
>   </property>
>   <property>
>      <name>mapred.fairscheduler.allocation.file</name>
>      <value>/etc/hadoop/conf/pools.xml</value>
>   </property>
>   <property>
>      <name>mapred.fairscheduler.assignmultiple</name>
>      <value>true</value>
>   </property>
>   <property>
>      <name>mapred.fairscheduler.poolnameproperty</name>
>      <value>pool.name</value>
>   </property>
>   <property>
>      <name>pool.name</name>
>      <value>default</value>
>   </property>
>
> </configuration>
>
> and /etc/hadoop/conf/pools.xml
>
> <?xml version="1.0"?>
> <allocations>
>   <pool name="realtime">
>     <minMaps>4</minMaps>
>     <minReduces>1</minReduces>
>     <minSharePreemptionTimeout>180</minSharePreemptionTimeout>
>     <weight>2.0</weight>
>   </pool>
>   <pool name="default">
>     <minMaps>2</minMaps>
>     <minReduces>2</minReduces>
>     <maxRunningJobs>1</maxRunningJobs>
>   </pool>
> </allocations>
>
> but a job in the realtime pool fails to interrupt a job running in the
> default queue (waited for > 15 minutes).  Is there something wrong with my
> configs?  Or is there anything in the logs that would be useful for
> debugging?  (I've only found a "successfully configured fairscheduler"
> comment in the jobtracker log upon starting up the daemon.)
>
> Help would be extremely appreciated!
>
> Thanks,
> -James Warren
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message