hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4439) Cleanup memory related resource management
Date Fri, 17 Oct 2008 08:22:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640465#action_12640465

Hemanth Yamijala commented on HADOOP-4439:

The following changes are identified:

- We remove the concept of having a default memory per task on the TT, introduced in HADOOP-3759
as the max memory per TT / number of slots. The problem with this model is that in a heterogenous
cluster, different TTs could give different default memory per task values for the same job,
which is confusing.
- Instead, we introduce a default memory per task configuration variable that is expected
to be controlled by the cluster admin. This is the value that will be used for a job which
does not specify any memory requirements. The advantage with this model is that it eases configuration,
and makes the default value consistent for the users.
- If a job has not specified any memory requirements, this variable would be set to the job's
configuration, maybe via the {{Task}} object.
- We modify the algorithm of protecting RAM limits introduced in HADOOP-3581 to use the configured
memory per task instead of the default memory per task.
- We remove the reporting of the default memory per task, introduced in HADOOP-3759, done
via the {{TaskTrackerStatus.ResourceStatus}}. Instead we report the total memory available
on the TT instead of the default memory per task.
- When HADOOP-4053 is fixed, the above values would be used to schedule tasks.
- However until HADOOP-4053 is fixed, these configuration parameters and the corresponding
{{JobConf}} variables should not be exposed in any public API or documentation, as they could
confuse users. This can be turned on after Hadoop 0.19

> Cleanup memory related resource management
> ------------------------------------------
>                 Key: HADOOP-4439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4439
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.19.0
> HADOOP-3759 and HADOOP-3581 introduced memory based resource management. This JIRA is
to cleanup certain aspects of the two issues that came up while doing HADOOP-4035, which is
filed to support memory based scheduling 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message