hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maysam Yabandeh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2006) Estimate Job Endtime
Date Wed, 30 Apr 2014 14:47:14 GMT

    [ https://issues.apache.org/jira/browse/YARN-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985582#comment-13985582

Maysam Yabandeh commented on YARN-2006:

We built on top of the already existing estimator that the speculator uses to estimate the
task end time, and add the logic to estimate the job end time based on the estimated end time
of its tasks. The estimation has two steps: i) estimate the end time of the tasks that are
not run yet, ii) estimate the end time of the running tasks. For the former we reuse the already
existing logic in Speculator that estimates based on the mean of the previously executed tasks
(if there is any). For the later, we again reuse the already existing logic that estimates
the end time based on the current progress of the task attempts, and compute the minimum end
time between the concurrent attempts that are being run speculatively. The end time would
be the maximum end time of all the tasks. 

The overhead of estimation is O(tasks). To lower the overhead, we reuse the computed estimation
in the last call (j_prev_end) unless it is not valid anymore. Upon each progress report for
a task, we compare the task end-time estimation (t_end) with last estimation of its end time
1) if t_end > j_prev_end => j_end = t_end
2) if t_end < j_prev_end && t_end >= t_prev_end => j_end = j_prev_end
3) if t_end < j_prev_end && t_end < t_prev_end => j_end = ?
Only in case 3 we mark the job estimation invalid to be recomputed the next time on demand.

> Estimate Job Endtime
> --------------------
>                 Key: YARN-2006
>                 URL: https://issues.apache.org/jira/browse/YARN-2006
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Maysam Yabandeh
>            Assignee: Maysam Yabandeh
> YARN-1969 adds a new earliest-endtime-first policy to the fair scheduler. As a prerequisite
step, the AppMaster should estimate its end time and send it to the RM via the heartbeat.
This jira focuses on how the AppMaster performs this estimation.

This message was sent by Atlassian JIRA

View raw message