tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-2599) Dont send obsoleted data movement events to tasks
Date Thu, 02 Jul 2015 23:06:05 GMT
Bikas Saha created TEZ-2599:
-------------------------------

             Summary: Dont send obsoleted data movement events to tasks
                 Key: TEZ-2599
                 URL: https://issues.apache.org/jira/browse/TEZ-2599
             Project: Apache Tez
          Issue Type: Improvement
            Reporter: Bikas Saha
            Assignee: Bikas Saha


Since events are sent in bulk and in sequential order to tasks they can end up getting a bunch
of data movement events in round 1 and then an input failed event in round 2. The task may
end up scheduling fetches for these obsoleted events before round 2, leading to wastage. 
Given that an input failed event and data movement event can be matched via their source task
attempt ids, the AM can use this match to obsolete the data movements events ahead of time
and not send them to the tasks in the first place.
The input failed events still need to be sent to the tasks, so that they can obsolete any
data movements events that they may have received much earlier from the failed task attempt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message