hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
Date Wed, 09 Sep 2015 14:59:46 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736993#comment-14736993
] 

Sunil G commented on MAPREDUCE-5870:
------------------------------------

Thank you [~jlowe] for sharing the comments.

As discussed offline, I had doubts on couple of points here.

bq.We are already returning the job priority via getJobStatus
As I see, we are invoking {{YARNRunner#getJobStatus}} to get the job status while we try to
call {{Job#getPriority}}.
And in {{ClientServiceDelegate#getJobStatus}}, *JobStatus* is created from *JobReport* (from
AM).
{code}
  public JobStatus getJobStatus(JobID oldJobID) throws IOException {
    org.apache.hadoop.mapreduce.v2.api.records.JobId jobId =
      TypeConverter.toYarn(oldJobID);
    GetJobReportRequest request =
        recordFactory.newRecordInstance(GetJobReportRequest.class);
    request.setJobId(jobId);
    JobReport report = ((GetJobReportResponse) invoke("getJobReport",
        GetJobReportRequest.class, request)).getJobReport();
...
...
    jobStatus = TypeConverter.fromYarn(report, url);
...
{code}

As this JobReport doesnt have priority information, current code in {{TypeConverter#fromYarn}}
is to hardcode the status.

{code}
  public static JobStatus fromYarn(JobReport jobreport, String trackingUrl) {
    JobPriority jobPriority = JobPriority.NORMAL;
    JobStatus jobStatus = new org.apache.hadoop.mapred.JobStatus(
        fromYarn(jobreport.getJobId()), jobreport.getSetupProgress(), jobreport
....
{code}

AppReport has the information about the priority of the application. So a second call has
to be made to fetch the same from {{YarnRunner#getJobStatus}}. And it can be set to JobStatus
separately.
Kindly help to share your thoughts here if I am missing something important.

bq.Why was the call to testChangingJobPriority removed from TestMRJobClient?
{{TestMRJobClient#testChangingJobPriority}} was invoked after job was completed. And YARN
throws excpetion for change priority application if its not RUNNING or ACCEPTED. As updated
offline, I agree that YARN need not have to throw excpetion if the app is completed, rather
it can be skipped. I will file a ticket in YARN to fix the same. Thoughts?

For LocalJobRunner exception case, I will file another ticket to fix the exception and to
support correct priority.
Also I will add all possible test cases for Job and JobConf to see all possible integer and
enums are set/get correctly.
Thank you for the support.

> Support for passing Job priority through Application Submission Context in Mapreduce
Side
> -----------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5870
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: client
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, 0003-MAPREDUCE-5870.patch,
0004-MAPREDUCE-5870.patch, 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> 			a.	JobConf.getJobPriority() and Job.setPriority(JobPriority priority) 
> 			b.	We can also use configuration "mapreduce.job.priority".
> 		Now this Job priority can be passed in Application Submission context from Client side.
> 		Here we can reuse the MRJobConfig.PRIORITY configuration. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message