hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sami Dalouche <sa...@hopper.com>
Subject Capacity Scheduler : how to use more than the queue capacity ?
Date Thu, 20 Oct 2011 17:25:58 GMT
Hi,

By choosing the capacity scheduler, I was under the impression that each
queue could borrow other queues' resources if they are available.


Let's say we have the configuration below, and a total capacity of 180
slots.
What I expect is that whenever default and cpu-bound queues have no job,
then jobs submitted to io-bound should be able to borrow up to 90 slots (50%
total capacity).
However, it looks like it never gets above 59 slots (33% of 180 slots).

Is there something I missed ?
Thanks,
Sami Dalouche

---
<property>
    <name>mapred.capacity-scheduler.queue.default.capacity</name>
    <value>33</value>
  </property>
  <property>
      <name>mapred.capacity-scheduler.queue.default.maximum-capacity</name>
      <value>50</value>
    </property>
  <property>
    <name>mapred.capacity-scheduler.queue.default.supports-priority</name>
    <value>true</value>
  </property>

  <!-- queue: io-bound -->
  <property>
    <name>mapred.capacity-scheduler.queue.io-bound.capacity</name>
    <value>33</value>
  </property>
  <property>
      <name>mapred.capacity-scheduler.queue.io-bound.maximum-capacity</name>
      <value>50</value>
    </property>
  <property>
    <name>mapred.capacity-scheduler.queue.io-bound.supports-priority</name>
    <value>true</value>
  </property>

  <!-- queue: cpu-bound -->
  <property>
    <name>mapred.capacity-scheduler.queue.cpu-bound.capacity</name>
    <value>34</value>
  </property>
  <property>

<name>mapred.capacity-scheduler.queue.cpu-bound.maximum-capacity</name>
      <value>100</value>
    </property>
  <property>

<name>mapred.capacity-scheduler.queue.cpu-bound.supports-priority</name>
    <value>true</value>
  </property>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message