hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
Date Sat, 16 May 2020 01:11:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-23443?focusedWorklogId=433982&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433982
]

ASF GitHub Bot logged work on HIVE-23443:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 16/May/20 01:10
            Start Date: 16/May/20 01:10
    Worklog Time Spent: 10m 
      Work Description: prasanthj commented on a change in pull request #1012:
URL: https://github.com/apache/hive/pull/1012#discussion_r426102275



##########
File path: llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##########
@@ -884,10 +885,20 @@ private void finishableStateUpdated(TaskWrapper taskWrapper, boolean
newFinishab
         taskWrapper.updateCanFinishForPriority(newFinishableState);
         forceReinsertIntoQueue(taskWrapper, isRemoved);
       } else {
-        taskWrapper.updateCanFinishForPriority(newFinishableState);
-        if (!newFinishableState && !taskWrapper.isInPreemptionQueue()) {
-          // No need to check guaranteed here; if it was false we would already be in the
queue.
+        // if speculative task, any finishable state change should re-order the queue as
speculative tasks are always
+        // not-guaranteed (re-order helps put non-finishable's ahead of finishable)
+        if (!taskWrapper.isGuaranteed()) {
+          removeFromPreemptionQueue(taskWrapper);
+          taskWrapper.updateCanFinishForPriority(newFinishableState);
           addToPreemptionQueue(taskWrapper);
+        } else {
+          // if guaranteed task, if the finishable state changed to non-finishable and if
the task doesn't exist
+          // pre-emption queue, then add it so that it becomes candidate to kill
+          taskWrapper.updateCanFinishForPriority(newFinishableState);

Review comment:
       Non-finishable -> Finishable does not have to be pre-emption queue. This could be
in wait queue (if not capacity) or taken by executor to run both which are fine.
   You brought up good point, we may be adding the same task fragment to pre-emption queue
twice. I will add a if not exists check when adding to pre-emption queue. 
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 433982)
    Time Spent: 20m  (was: 10m)

> LLAP speculative task pre-emption seems to be not working
> ---------------------------------------------------------
>
>                 Key: HIVE-23443
>                 URL: https://issues.apache.org/jira/browse/HIVE-23443
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing pre-emption
to not work in certain cases.
> {code:java}
> "attempt_1589167813851_0000_119_01_000008_0 (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map
1, started at 2020-05-11 05:59:22, in preemption queue, can finish)", 
> "attempt_1589167813851_0008_84_01_000008_1 (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map
1, started at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is non-finishable. 
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not triggering pre-emption
queue re-ordering so peek() always returns canFinish task even though non-finishable tasks
are in the queue. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message