hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4018) limit memory usage in jobtracker
Date Fri, 05 Sep 2008 07:47:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628584#action_12628584
] 

Amar Kamat commented on HADOOP-4018:
------------------------------------

bq. I have a question regarding item 1 above. 
They are mutually exclusive. For data local tasks {{runningMapCache, noinRunningMapcache,
runningReduces and nonRunningReduces}} are used. For non-data local tasks {{nonLocalMaps and
nonLocalRunningMaps}} are used.
bq. it violates locking hierarchy, 
Yes. One thing you could do is keep a global count of all allocated tasks in the JobTracker.
Jobs getting constructed will check the value before bailing out. Once the job inits, it updates
the value of this count. Any access/update to the count should be guarded. Since the count
will be updates only on passing the init tests, we can be sure that limit-exceeding jobs never
get inited. So something like
{code}
In init :
  1. Get the count lock
  2. Check if the count + self-tasks > limit
      2.1 If yes then throw an exception
      2.2 else update the count and release the lock
{code}
Since the value of allocated tasks never change once inited, there is no point in iterating
over the jobs everytime. Once the job is removed from the jobtracker (see RetireJobs), then
update the count to reflect the change.

> limit memory usage in jobtracker
> --------------------------------
>
>                 Key: HADOOP-4018
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4018
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: maxSplits.patch, maxSplits2.patch, maxSplits3.patch, maxSplits4.patch,
maxSplits5.patch, maxSplits6.patch
>
>
> We have seen instances when a user submitted a job with many thousands of mappers. The
JobTracker was running with 3GB heap, but it was still not enough to prevent memory trashing
from Garbage collection; effectively the Job Tracker was not able to serve jobs and had to
be restarted.
> One simple proposal would be to limit the maximum number of tasks per job. This can be
a configurable parameter. Is there other things that eat huge globs of memory in job Tracker?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message