hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3759) Provide ability to run memory intensive jobs without affecting other running tasks on the nodes
Date Wed, 23 Jul 2008 18:29:33 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Hemanth Yamijala updated HADOOP-3759:

    Attachment: HADOOP-3759.patch

Initial patch for review and to encourage discussion

The patch incorporates the approach mentioned in the earlier comments. Specifically:

- Includes the configuration variables described and defines accessors in JobConf

- Includes a Map for storing resource information in TaskTrackerStatus. Though there is only
one entry currently (free memory), this map is an attempt to keep the wire protocol the same
even in future when more resources are to be added. I've currently used the value of these
resources as Long, but possibly this should be changed to a Writable so we can pass anything

- Defines a method to compute minimum free space that would be available for a new task in
TaskTracker (using the configuration variables for currently running tasks), and setting that
value in TaskTrackerStatus.

- Accessors in JobInProgress to define memory requirements for the job.

With these changes in place, schedulers such as HADOOP-3445 can decide to  schedule tasks
according to the job's memory requirements, and the free memory available on a tasktracker.

Please provide feedback on this implementation.

> Provide ability to run memory intensive jobs without affecting other running tasks on
the nodes
> -----------------------------------------------------------------------------------------------
>                 Key: HADOOP-3759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3759
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.19.0
>         Attachments: HADOOP-3759.patch
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from affecting
Hadoop daemons and other tasks running on a node. A related requirement is that users be provided
an ability to run jobs which are memory intensive. The system must provide enough knobs to
allow such jobs to be run while still maintaining the requirements of HADOOP-3581.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message