hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From james warren <ja...@rockyou.com>
Subject fair scheduler preemptions timeout difficulties
Date Thu, 03 Dec 2009 00:26:23 GMT
Greetings, Hadoop Fans:

I'm attempting to use the timeout feature of the Fair Scheduler (using
Cloudera's most recently released distribution 0.20.1+152-1), but without
success.  I'm using the following configs:

/etc/hadoop/conf/mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>hadoop-master:8021</value>
  </property>
  <property>
     <name>mapred.tasktracker.map.tasks.maximum</name>
     <value>9</value>
  </property>
  <property>
     <name>mapred.tasktracker.reduce.tasks.maximum</name>
     <value>3</value>
  </property>
  <property>
     <name>mapred.jobtracker.taskScheduler</name>
     <value>org.apache.hadoop.mapred.FairScheduler</value>
  </property>
  <property>
     <name>mapred.fairscheduler.allocation.file</name>
     <value>/etc/hadoop/conf/pools.xml</value>
  </property>
  <property>
     <name>mapred.fairscheduler.assignmultiple</name>
     <value>true</value>
  </property>
  <property>
     <name>mapred.fairscheduler.poolnameproperty</name>
     <value>pool.name</value>
  </property>
  <property>
     <name>pool.name</name>
     <value>default</value>
  </property>

</configuration>

and /etc/hadoop/conf/pools.xml

<?xml version="1.0"?>
<allocations>
  <pool name="realtime">
    <minMaps>4</minMaps>
    <minReduces>1</minReduces>
    <minSharePreemptionTimeout>180</minSharePreemptionTimeout>
    <weight>2.0</weight>
  </pool>
  <pool name="default">
    <minMaps>2</minMaps>
    <minReduces>2</minReduces>
    <maxRunningJobs>1</maxRunningJobs>
  </pool>
</allocations>

but a job in the realtime pool fails to interrupt a job running in the
default queue (waited for > 15 minutes).  Is there something wrong with my
configs?  Or is there anything in the logs that would be useful for
debugging?  (I've only found a "successfully configured fairscheduler"
comment in the jobtracker log upon starting up the daemon.)

Help would be extremely appreciated!

Thanks,
-James Warren

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message