hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3759) Provide ability to run memory intensive jobs without affecting other running tasks on the nodes
Date Wed, 06 Aug 2008 04:08:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620129#action_12620129
] 

Vivek Ratan commented on HADOOP-3759:
-------------------------------------

I'm in favor of the approach Hemanth has described above because it lets us reason about memory
constraints in terms of slots/containers. In his example, each slot on the TT has a memory
limit of 4GB. This also means that a task is guaranteed up to 4GB, if it runs in that slot.
So if  a task needs 6GB, it ends up using 2 slots. This helps schedulers, because they're
really dealing with slots. For example, if they're keeping track of how many slots are used
by a user or job, then assigning two slots to the task that wants 6GB works really well. 

Another good thing about this approach is that tasks are not affected by other tasks, in terms
of memory guarantees. Suppose task1 wanted 7.5 GB. This would leave 0.5GB for task2, which
is not right, as task2 shouldn't suffer from task1's needs. It's better task2 not run in this
case. 

> Provide ability to run memory intensive jobs without affecting other running tasks on
the nodes
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3759
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.19.0
>
>         Attachments: HADOOP-3759.patch
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from affecting
Hadoop daemons and other tasks running on a node. A related requirement is that users be provided
an ability to run jobs which are memory intensive. The system must provide enough knobs to
allow such jobs to be run while still maintaining the requirements of HADOOP-3581.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message