hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5083) Optionally a separate daemon should serve JobHistory
Date Fri, 30 Jan 2009 06:29:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668765#action_12668765

dhruba borthakur commented on HADOOP-5083:

Amar, thanks for the explanation.

I am assuming that the history folder will have lots and lots of jobs over time (possibly
in the tens of thousands). In that case, when a user submits a job in the morning and then
comes back the following day to look at the status of his job. he/she finds that the job is
retired. he/she then goes to the history server. Now, he/she has to locate his job among the
tens of thousands of completed jobs that are in the history folder. The user, typically, does
not remember his job id. What does the user do now? Does the history server allow the user
to retrieve a list of jobs that he/she submitted? Does it allow a user to list jobs based
on reverse chronological order? Just asking :-)

> Optionally a separate daemon should serve JobHistory
> ----------------------------------------------------
>                 Key: HADOOP-5083
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5083
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>            Assignee: Amar Kamat
>         Attachments: HADOOP-5083-v1.2.patch, HADOOP-5083-v1.9.patch
> Currently the JobTracker serves the JobHistory to end-users off files local-disk/hdfs.
While running very large clusters with a large user-base might result in lots of traffic for
job-history which needlessly taxes the JobTracker. The proposal is to have an optional daemon
which handles serving of job-history requests.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message