tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohini Palaniswamy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-394) Handle uneven DAGs
Date Fri, 23 Aug 2013 19:19:51 GMT
Rohini Palaniswamy created TEZ-394:

             Summary: Handle uneven DAGs
                 Key: TEZ-394
                 URL: https://issues.apache.org/jira/browse/TEZ-394
             Project: Apache Tez
          Issue Type: Sub-task
            Reporter: Rohini Palaniswamy

  Consider a series of joins or group by on dataset A with few datasets that takes 10 hours
followed by a final join with a dataset X. The vertex that loads dataset X will be one of
the top vertexes and initialized early even though its output is not consumed till the end
after 10 hours. 

1) Could either use delayed start logic for better resource allocation
2) Else if they are started upfront, need to handle failure/recovery cases where the nodes
which executed the MapTask might have gone down when the final join happens. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message