hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal Vijayaraghavan (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-22966) LLAP: Consider including waitTime for comparing attempts in same vertex
Date Thu, 05 Mar 2020 17:51:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-22966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052380#comment-17052380
] 

Gopal Vijayaraghavan commented on HIVE-22966:
---------------------------------------------

bq. even thought this patch takes into account task aging we do not cure the long-tail task
issue and we need to properly take care of it.

This entire patch is hiding in the shadow of YARN FIFO assumptions in long tail task scheduling
order code inside Tez.

https://github.com/apache/tez/blob/master/tez-runtime-library/src/main/java/org/apache/tez/dag/library/vertexmanager/ShuffleVertexManager.java#L591

There's also a somewhat equivalent version for the splits as well

https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/MRInputHelpers.java#L501

So Tez explicitly picks the biggest splits and the heaviest skewed reducers to start first,
which is mostly relevant for query latency when we have a large number of tasks and a low
number of executors.

That is why this patch makes a difference, because at the same priority, we get FIFO back.

> LLAP: Consider including waitTime for comparing attempts in same vertex
> -----------------------------------------------------------------------
>
>                 Key: HIVE-22966
>                 URL: https://issues.apache.org/jira/browse/HIVE-22966
>             Project: Hive
>          Issue Type: Improvement
>          Components: llap
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Minor
>             Fix For: 4.0.0
>
>         Attachments: HIVE-22966.3.patch, HIVE-22966.4.patch
>
>
> When attempts are compared within same vertex, it should pick up the attempt with longest
wait time to avoid starvation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message