hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4039) Retired jobs are not present in the job list returned to the job-client.
Date Tue, 02 Sep 2008 13:29:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627670#action_12627670

Vinod Kumar Vavilapalli commented on HADOOP-4039:

May be we better also have a parameter N saying we only need the list of last N jobs.

I am seeing a common pattern here - get the list of jobs that are yet to complete (JobSubmissionProtocol.jobsToComplete),
get the list of running jobs, queued jobs, jobs that have started in the last N hours, last
N jobs etc. We have two options here:
 - Get all jobs list from the JT and prune them on the client side.
 - Pass a filter to the JT asking for only those jobs that we need.

When the number of jobs could get very long, the second option seems better - less data from
JT to client, accomodates all the above types of listing jobs in a single RPC, for e.g., we
then won't need a separate jobsToComplete. Thoughts?

> Retired jobs are not present in the job list returned to the job-client.
> ------------------------------------------------------------------------
>                 Key: HADOOP-4039
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4039
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
> After {{mapred.jobtracker.retirejob.interval}} elapses, completed jobs are no longer
maintained by the JT, but instead when job-client ask for job status, counters or task completion
events, the relevant information is picked up from completed job store. But a retired job
is not listed in the output of "{{hadoop job -list all}}", without which other information
from completed job store isn't quite useful unless a job-id is known from elsewhere.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message