tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-348) Improve how completion events are fetched by the ShuffleHandler
Date Fri, 23 Aug 2013 22:11:52 GMT

    [ https://issues.apache.org/jira/browse/TEZ-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13749070#comment-13749070

Bikas Saha commented on TEZ-348:

For option 2, the number of connections is not different from the case where the AM receives
incoming connections from tasks via pull-based umbilical, right?

If we change the tasks to start a light-weight custom (netty) HTTP server to which the AM
could post a PB buffer payload then we would that be a viable solution?
> Improve how completion events are fetched by the ShuffleHandler
> ---------------------------------------------------------------
>                 Key: TEZ-348
>                 URL: https://issues.apache.org/jira/browse/TEZ-348
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Hitesh Shah
> The ShuffleHandler currently has a thread which checks for new completion events every
second. This can add unnecessary delay to the reduce getting started.
> An async RPC implementation would work well to fix this. That, however, is currently
not available in Hadoop.
> Options
> - Poll with a smaller interval. This can overload the AM if there's a large number of
reduce tasks. The poll interval could be set based on the # of tasks.
> - Have the AM push completion events to the Task. AM ends up creating way too many connections,
and the child has to run an RPC server.
> - Rely on an external service like ZK with monitors.
> Thoughts / suggestions on how this can be improved ?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message