hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3759) Provide ability to run memory intensive jobs without affecting other running tasks on the nodes
Date Wed, 13 Aug 2008 17:20:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622284#action_12622284
] 

Hemanth Yamijala commented on HADOOP-3759:
------------------------------------------

Ari, I had looked at the ResourceEstimator class that you defined in HADOOP-657. In this patch
we are not estimating the used memory - rather using a configured value. If we go to a model
where we estimate memory per task, then we should definitely use the ResourceEstimator. But
for now, we are going with a simpler approach of letting the user specify this to us in configuration.

On the other hand, to communicate the used disk space, you are following roughly the same
mechanism as what is defined here IIRC - which is to compute the free disk space in the heartbeat
and communicate it via the TaskTrackerStatus. Vinod's comment was to unify your change as
part of the resource map defined here so that it would be done in a like manner.

Makes sense ?

> Provide ability to run memory intensive jobs without affecting other running tasks on
the nodes
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3759
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.19.0
>
>         Attachments: HADOOP-3759.patch, HADOOP-3759.patch, HADOOP-3759.patch
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from affecting
Hadoop daemons and other tasks running on a node. A related requirement is that users be provided
an ability to run jobs which are memory intensive. The system must provide enough knobs to
allow such jobs to be run while still maintaining the requirements of HADOOP-3581.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message