tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-2612) Critical path analyzer for DAGs
Date Fri, 10 Jul 2015 13:07:04 GMT
Rajesh Balamohan created TEZ-2612:

             Summary: Critical path analyzer for DAGs
                 Key: TEZ-2612
                 URL: https://issues.apache.org/jira/browse/TEZ-2612
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Rajesh Balamohan

This analyzer plugin/tool can be used to understand the important vertices/tasks of interest
in large DAG for perf analysis / finding bottlenecks.  It can be used to find out 1.input
dependency, 2. failure dependency, 3. scheduling dependency (may be at later stage).

Creating this as a uber ticket. Getting this detail at vertex level might be possible with
the existing logs derived from ATS.  For task level analysis, certain more details are required.

1. Timeline details like when fetch/merge/compute/sort etc are not captured now. These details
can possibly be added in TaskCompletionEvent
2. Need additional details like the last event that completed processing in the input (for
tracing at task level)
3. Add downstream task attempt that caused the higher level task to get rescheduled/restarted.
This can be used in terms of understanding in cases where the task failed due to read-error.

This message was sent by Atlassian JIRA

View raw message