tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-348) Improve how completion events are fetched by the ShuffleHandler
Date Tue, 06 Aug 2013 17:12:50 GMT
Siddharth Seth created TEZ-348:
----------------------------------

             Summary: Improve how completion events are fetched by the ShuffleHandler
                 Key: TEZ-348
                 URL: https://issues.apache.org/jira/browse/TEZ-348
             Project: Apache Tez
          Issue Type: Improvement
            Reporter: Siddharth Seth


The ShuffleHandler currently has a thread which checks for new completion events every second.
This can add unnecessary delay to the reduce getting started.
An async RPC implementation would work well to fix this. That, however, is currently not available
in Hadoop.

Options
- Poll with a smaller interval. This can overload the AM if there's a large number of reduce
tasks. The poll interval could be set based on the # of tasks.
- Have the AM push completion events to the Task. AM ends up creating way too many connections,
and the child has to run an RPC server.
- Rely on an external service like ZK with monitors.

Thoughts / suggestions on how this can be improved ?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message