hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5083) Optionally a separate daemon should serve JobHistory
Date Wed, 28 Jan 2009 18:35:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668103#action_12668103
] 

dhruba borthakur commented on HADOOP-5083:
------------------------------------------

We are 0.17 (and 0.19) with long-running JTs and one problem we are seeing is that the JT
sometimes runs out of its 3GB heap space. The system is used by around 50-80 users. The max
jobs per user before retirement is 5. But this still means that the JT keeps info about 80*5=400
completed jobs in memory. Sometimes these jobs have huge number of tasks. This eats up most
of the memory in the JT.  

Can part of this fix (i.e. purge all jobs from completed queue when there is memory usage
on the JT exceeds a configured threshold) be ported to 0.19. It is very useful to have it
in 0.19, otherwise the JT just hangs up and the cluster has to be restarted. 

> Optionally a separate daemon should serve JobHistory
> ----------------------------------------------------
>
>                 Key: HADOOP-5083
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5083
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>            Assignee: Amar Kamat
>         Attachments: HADOOP-5083-v1.2.patch, HADOOP-5083-v1.9.patch
>
>
> Currently the JobTracker serves the JobHistory to end-users off files local-disk/hdfs.
While running very large clusters with a large user-base might result in lots of traffic for
job-history which needlessly taxes the JobTracker. The proposal is to have an optional daemon
which handles serving of job-history requests.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message