hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4974) Make jobtracker resilient to memory issues
Date Fri, 02 Jan 2009 08:51:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660276#action_12660276

Amar Kamat commented on HADOOP-4974:

As of now these are the cases where the jobtracker should guard itself :
- *JOB INIT :* Consider a case when the jobtracker is running low on memory. Ideally we expect
the jobtracker to queue(not expand) jobs that might potentially bring down the jobtracker.

-  *JSP :* _loadhistory.jsp_ caches job analysis results. For larger jobs this can occupy
some amount of memory. Job history parsing results are cached because these results are required
for  fine-grained analysis (like task analysis and job analysis). Parsing the job history
again and again for every tip will be costly. The problem occurs when the jobtracker runs
low on memory and we try to analyze a job from history.
- *OLD JOBS :* Look at HADOOP-4766 for more details.

> Make jobtracker resilient to memory issues
> ------------------------------------------
>                 Key: HADOOP-4974
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4974
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.21.0
>            Reporter: Amar Kamat
>             Fix For: 0.21.0
> JobTracker is vulnerable to memory errors/attacks. Few of them are as follows
> - *JOB INIT :* lot of users submitting large jobs. As every jobs is expanded, the jobtracker's
memory can be completely used up
> - *JSP :* jsp (jobhistory.jsp etc) can also interfere with jobtracker's memory and hence
the jobtracker should be protected against such attacks
> - *OLD JOBS :* lot of completed jobs can garble up jobtracker's memory and hence should
be periodically cleaned up. HADOOP-4766 addresses this.
> The main intention of this issue is to track various jira's that help jobtracker battle
memory attacks. Jobtracker should always be up and available. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message