hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3759) Provide ability to run memory intensive jobs without affecting other running tasks on the nodes
Date Tue, 15 Jul 2008 12:23:31 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613596#action_12613596

Hemanth Yamijala commented on HADOOP-3759:

Initial proposal:

- HADOOP-3581 proposes a maximum amount of virtual memory, say MAX_MEM,  that all tasks (and
their descendants) on that tasktracker would use.
- By default, we can translate this to a per task memory limit which is = MAX_MEM / number
of slots
- To allow high memory jobs to run, we define a new configuration variable that users can
set to specify the maximum memory they expect their tasks to take.
- In each heartbeat, the tasktracker computes the amount of free memory (using HADOOP-3581's
fix) and reports that to the jobtracker. This is similar to the approach followed in HADOOP-657
for disk space.
- The jobtracker schedules a new task on this tasktracker, only if it's job's memory limit
per task is less than the free memory.
- If a job's task takes more memory than what is specified (either default or explicitly configured),
per HADOOP-3581, it will be killed.

Comments ?

> Provide ability to run memory intensive jobs without affecting other running tasks on
the nodes
> -----------------------------------------------------------------------------------------------
>                 Key: HADOOP-3759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3759
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.19.0
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from affecting
Hadoop daemons and other tasks running on a node. A related requirement is that users be provided
an ability to run jobs which are memory intensive. The system must provide enough knobs to
allow such jobs to be run while still maintaining the requirements of HADOOP-3581.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message