hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5083) Optionally a separate daemon should serve JobHistory
Date Thu, 29 Jan 2009 08:21:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668369#action_12668369
] 

Amar Kamat commented on HADOOP-5083:
------------------------------------

bq. We are 0.17 (and 0.19) with long-running JTs and one problem we are seeing is that the
JT sometimes runs out of its 3GB heap space. The system is used by around 50-80 users. The
max jobs per user before retirement is 5. But this still means that the JT keeps info about
80*5=400 completed jobs in memory. Sometimes these jobs have huge number of tasks. This eats
up most of the memory in the JT.
Wouldnt making the number of completed jobs in memory = 0 help? But the problem is that the
completed jobs will be available only via history and that will risk the jobtracker.

bq. I was wondering if ay type of fix for this problem is going to be checked into the 019
branch?
I think it will be big change to go in 0.19, no? Comments?

> Optionally a separate daemon should serve JobHistory
> ----------------------------------------------------
>
>                 Key: HADOOP-5083
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5083
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>            Assignee: Amar Kamat
>         Attachments: HADOOP-5083-v1.2.patch, HADOOP-5083-v1.9.patch
>
>
> Currently the JobTracker serves the JobHistory to end-users off files local-disk/hdfs.
While running very large clusters with a large user-base might result in lots of traffic for
job-history which needlessly taxes the JobTracker. The proposal is to have an optional daemon
which handles serving of job-history requests.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message