hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-5567) Add support for scheduling jobs based on memory requirements to the Fairscheduler
Date Wed, 25 Mar 2009 04:23:50 GMT
Add support for scheduling jobs based on memory requirements to the Fairscheduler

                 Key: HADOOP-5567
                 URL: https://issues.apache.org/jira/browse/HADOOP-5567
             Project: Hadoop Core
          Issue Type: New Feature
          Components: contrib/fair-share
            Reporter: Hemanth Yamijala

In HADOOP-3759, we added the ability for users to specify jobs requesting for a certain amount
of virtual memory. For e.g. users can say that their jobs require 2GB of memory to run. In
HADOOP-4035, functionality was added to the capacity scheduler to schedule jobs based on this
specified amount. This JIRA is to add similar support to the Fairshare scheduler.

The basic use case is that there are jobs that require a certain known amount of virtual memory,
usually more than the JVM's heap size. This happens specifically for streaming jobs that can
launch several processes from the child. Without being aware of these requirements, if tasks
are scheduled on nodes just based on available slots, they have a potential of affecting the
other processes running on the node, or if memory protection features are enabled (HADOOP-3581),
they could result in the task being killed by the tasktracker.

The scheduler must take into account the requested amount of memory by the job, the amount
of memory that can be committed to by a tracker, and schedule based on these inputs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message