tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-2599) Dont send obsoleted data movement events to tasks
Date Sat, 04 Jul 2015 08:12:04 GMT

    [ https://issues.apache.org/jira/browse/TEZ-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613600#comment-14613600

Bikas Saha commented on TEZ-2599:

bq. edgeManagerOnDemand.routeInputSourceTaskFailedEventToDestination' itself as the pendingRoute
will anyways be null for INPUT_FAILED_EVENT
It may not be null because they can be expanded too (e.g. custom shuffle edge manager) 

Thanks for the review. Committing latest patch that has a test fix. Ran tests locally since
pre-commit seems to be down.

This only adds an if check on the event to check for obsoletion before sending it.

> Dont send obsoleted data movement events to tasks
> -------------------------------------------------
>                 Key: TEZ-2599
>                 URL: https://issues.apache.org/jira/browse/TEZ-2599
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>         Attachments: TEZ-2599.1.patch, TEZ-2599.2.patch
> Since events are sent in bulk and in sequential order to tasks they can end up getting
a bunch of data movement events in round 1 and then an input failed event in round 2. The
task may end up scheduling fetches for these obsoleted events before round 2, leading to wastage.

> Given that an input failed event and data movement event can be matched via their source
task attempt ids, the AM can use this match to obsolete the data movements events ahead of
time and not send them to the tasks in the first place.
> The input failed events still need to be sent to the tasks, so that they can obsolete
any data movements events that they may have received much earlier from the failed task attempt.

This message was sent by Atlassian JIRA

View raw message