Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <441955631.1233296999643.JavaMail.jira@brutus>
Date: Thu, 29 Jan 2009 22:29:59 -0800 (PST)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5083) Optionally a separate daemon should
 serve JobHistory
In-Reply-To: <644178812.1232443799608.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668765#action_12668765 ] 

dhruba borthakur commented on HADOOP-5083:
------------------------------------------

Amar, thanks for the explanation.

I am assuming that the history folder will have lots and lots of jobs over time (possibly in the tens of thousands). In that case, when a user submits a job in the morning and then comes back the following day to look at the status of his job. he/she finds that the job is retired. he/she then goes to the history server. Now, he/she has to locate his job among the tens of thousands of completed jobs that are in the history folder. The user, typically, does not remember his job id. What does the user do now? Does the history server allow the user to retrieve a list of jobs that he/she submitted? Does it allow a user to list jobs based on reverse chronological order? Just asking :-)

> Optionally a separate daemon should serve JobHistory
> ----------------------------------------------------
>
>                 Key: HADOOP-5083
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5083
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>            Assignee: Amar Kamat
>         Attachments: HADOOP-5083-v1.2.patch, HADOOP-5083-v1.9.patch
>
>
> Currently the JobTracker serves the JobHistory to end-users off files local-disk/hdfs. While running very large clusters with a large user-base might result in lots of traffic for job-history which needlessly taxes the JobTracker. The proposal is to have an optional daemon which handles serving of job-history requests.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.