hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6916) History server scheduling tasks at fixed rate can be problematic when those tasks are slow
Date Tue, 18 Jul 2017 20:59:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092179#comment-16092179

Jason Lowe commented on MAPREDUCE-6916:

Slow or temporarily unavailable namenodes and slow datanodes are examples of things that can
cause one or both of these tasks to execute more slowly than it should.  We want to run these
tasks at a fixed rate, but we don't want to "catch up" any missing task executions if we fall
behind since there's no point to that.  A better design would be to have the tasks that are
actually scheduled with the executor to not do the full job but simply notify another thread
that is responsible for doing the task.  That way the executor tasks are quick and should
never back up in practice, and we won't end up double-executing if we notify more than once
while the task is still running.

> History server scheduling tasks at fixed rate can be problematic when those tasks are
> ------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-6916
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6916
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>    Affects Versions: 2.7.4
>            Reporter: Jason Lowe
> The job history server currently schedules both the task of moving jobs from intermediate
to done and the task of cleaning jobs at a fixed rate.  If those tasks take longer than the
rate period to execute then a backlog of to-be-scheduled tasks can build up and cause a long
storm of them to execute later when the blockage clears.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message