hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erix Yao <yao.e...@gmail.com>
Subject Questions about Fair scheduler in hadoop
Date Tue, 26 Apr 2011 16:21:15 GMT
hi,
I have 8 machines for the hadoop cluster, 1 namenode and 7 data node.
I want the production jobs to have more priority than the user-defined jobs,
so I use the Fair scheduler.

Why sometimes my job scheduled by user: hadoop just start 7 map tasks, while
 in the cluster there's no other job running or waiting to be run?

Here's my configuration section in mapred-site.xml:
<property>
  <name>mapred.jobtracker.taskScheduler</name>
  <value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
 <property>
<name>mapred.fairscheduler.allocation.file</name>
<value>conf/pools.xml</value>
</property>

and here's the pool.xml configuration:
<?xml version="1.0"?>
<allocations>
  <pool name="hadoop">
    <minMaps>30</minMaps>
    <minReduces>30</minReduces>
    <weight>4.0</weight>
  </pool>
  <user name="hive">
    <maxRunningJobs>20</maxRunningJobs>
  </user>
  <userMaxJobsDefault>10</userMaxJobsDefault>
</allocations>

thanks!



-- 
haitao.yao@Beijing

Mime
View raw message