hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Date Thu, 01 Dec 2011 22:58:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161270#comment-13161270
] 

Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------

@Mahedev and others

A follow up question. In the MR1 code, there was a way to assign a priority to a specific
job. This let users whose jobs all go to the same queue give more importance to certain jobs
over others and it is used in the fair scheduler.

The MR2 code does seem to have similar support, i.e. per-AM priorities. For instance, in ApplicationSubmissionContext
we see:

{noformat} 
  /**
   * Set the <code>Priority</code> of the application.
   * @param priority <code>Priority</code> of the application
   */
  @Public
  @Stable
  public void setPriority(Priority priority);
{noformat} 

The MR AM also seems to support per-job prioritization, though it doesn't look like this code
is called anywhere.
{noformat}
  /**
   * Set the priority of a running job.
   * @param priority the new priority for the job.
   * @throws IOException
   */
  public void setPriority(JobPriority priority) 
      throws IOException, InterruptedException {
{noformat}

Such priorities are *not*, however, passed downstream to the scheduler as far as I can see
(they aren't reflected in SchedulerApp class). I can just patch the code to pass these to
the scheduler, but I'm curious whether there is something I'm missing here, especially given
the last comment which seems to contradict that priorities can be applied across jobs.
                
> Port Fair Scheduler to MR2
> --------------------------
>
>                 Key: MAPREDUCE-3451
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Patrick Wendell
>            Assignee: Patrick Wendell
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to
MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include
design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message