hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4018) limit memory usage in jobtracker
Date Fri, 29 Aug 2008 04:44:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12626842#action_12626842
] 

Devaraj Das commented on HADOOP-4018:
-------------------------------------

Sorry Dhruba, I somehow missed the jira mail with your earlier comment. 

I think we should do this in the server (if at all we want to do it) since this should be
a admin configured param. Also proper warning should be given as response to the user if the
JT discards a job because of too many splits.. Maybe we should maintain a global count of
the tasks across all jobs as well and if that is exceeded we don't accept any job till some
job(s) complete..

By the way, configuring the mapred.jobtracker.completeuserjobs.maximum to a lower value should
help your case.

> limit memory usage in jobtracker
> --------------------------------
>
>                 Key: HADOOP-4018
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4018
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: maxSplits.patch
>
>
> We have seen instances when a user submitted a job with many thousands of mappers. The
JobTracker was running with 3GB heap, but it was still not enough to prevent memory trashing
from Garbage collection; effectively the Job Tracker was not able to serve jobs and had to
be restarted.
> One simple proposal would be to limit the maximum number of tasks per job. This can be
a configurable parameter. Is there other things that eat huge globs of memory in job Tracker?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message