quickstep-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jianqiao <...@git.apache.org>
Subject [GitHub] incubator-quickstep pull request #76: Visualize execution plan DAGs annotate...
Date Wed, 03 Aug 2016 17:13:58 GMT
GitHub user jianqiao opened a pull request:

    https://github.com/apache/incubator-quickstep/pull/76

    Visualize execution plan DAGs annotated with profiling stats

    This PR adds support for visualization of execution plan DAGs. Each relational operator
in the DAG is also annotated with the execution profiling stats of its work orders. The main
changes are
    - Added a `getName()` method for each relational operator.
    - Implemented visualization code in `utility/ExecutionDAGVisualizer.*`.
    - Slightly refactored work order profiling code to record both start time and end time
of each work order.
    
    To enable this feature, set the gflag `visualize_execution_dag` to true, e.g.
    ```
    quickstep_cli_shell -visualize_execution_dag=true
    ```
    The execution dag will be printed via `stderr` in DOT format. Each relational operator
node contains the following information:
    - Relational operator ID, Relational operator name.
    - The input relation's name (if it is a stored relation).
    - Total execution time of all the work orders of this relational operator.
    - Execution time span (start time of the first work order and finish time of the last
work order) of this relational operator.
    - Degree of concurrency (total time / span) of this relational operator.
    
    Here's an example output, for SSB query 05:
    ```
    digraph g {
      rankdir=BT
      node [penwidth=2]
      edge [fontsize=16 fontcolor=gray penwidth=2]
    
      0 [ label="[0] SelectOperator&#10;input stored relation [supplier]&#10;span:
[0ms, 6ms] (0.74%)&#10;total: 38ms (0.13%)&#10;effective concurrency: 6.29" style=filled
fillcolor="0.004717 0.004717 1.0" ]
      1 [ label="[1] SelectOperator&#10;input stored relation [part]&#10;span: [0ms,
10ms] (1.25%)&#10;total: 273ms (0.90%)&#10;effective concurrency: 26.26" style=filled
fillcolor="0.011673 0.011673 1.0" ]
      2 [ label="[2] BuildHashOperator&#10;span: [10ms, 12ms] (0.21%)&#10;total: 19ms
(0.06%)&#10;effective concurrency: 11.20" style=filled fillcolor="0.001494 0.001494 1.0"
]
      3 [ label="[3] HashJoinOperator&#10;probe side stored relation [lineorder]&#10;span:
[17ms, 752ms] (88.04%)&#10;total: 29059ms (95.43%)&#10;effective concurrency: 39.53"
style=filled fillcolor="1.000000 1.000000 1.0" ]
      5 [ label="[5] BuildHashOperator&#10;span: [6ms, 9ms] (0.39%)&#10;total: 6ms
(0.02%)&#10;effective concurrency: 1.97" style=filled fillcolor="0.002248 0.002248 1.0"
]
      6 [ label="[6] HashJoinOperator&#10;span: [748ms, 773ms] (3.04%)&#10;total:
438ms (1.44%)&#10;effective concurrency: 17.28" style=filled fillcolor="0.024394 0.024394
1.0" ]
      8 [ label="[8] BuildHashOperator&#10;input stored relation [ddate]&#10;span:
[8ms, 8ms] (0.05%)&#10;total: 0ms (0.00%)&#10;effective concurrency: 1.00" style=filled
fillcolor="0.000274 0.000274 1.0" ]
      9 [ label="[9] HashJoinOperator&#10;span: [767ms, 806ms] (4.65%)&#10;total:
372ms (1.22%)&#10;effective concurrency: 9.59" style=filled fillcolor="0.032002 0.032002
1.0" ]
      11 [ label="[11] AggregationOperator&#10;span: [792ms, 832ms] (4.80%)&#10;total:
223ms (0.73%)&#10;effective concurrency: 5.58" style=filled fillcolor="0.030163 0.030163
1.0" ]
      12 [ label="[12] FinalizeAggregationOperator&#10;span: [832ms, 833ms] (0.02%)&#10;total:
0ms (0.00%)&#10;effective concurrency: 1.00" style=filled fillcolor="0.000139 0.000139
1.0" ]
      13 [ label="[13] SortRunGenerationOperator&#10;span: [834ms, 834ms] (0.01%)&#10;total:
0ms (0.00%)&#10;effective concurrency: 1.00" style=filled fillcolor="0.000030 0.000030
1.0" ]
      14 [ label="[14] SortMergeRunOperator&#10;span: [834ms, 834ms] (0.01%)&#10;total:
0ms (0.00%)&#10;effective concurrency: 1.00" style=filled fillcolor="0.000043 0.000043
1.0" ]
      16 [ label="[16] SelectOperator&#10;span: [834ms, 834ms] (0.00%)&#10;total:
0ms (0.00%)&#10;effective concurrency: 1.00" style=filled fillcolor="0.000011 0.000011
1.0" ]
    
      0 -> 6 [ style=dashed ]
      0 -> 5 [ ]
      1 -> 3 [ style=dashed ]
      1 -> 2 [ ]
      2 -> 3 [ style=dashed ]
      3 -> 6 [ ]
      5 -> 6 [ style=dashed ]
      6 -> 9 [ ]
      8 -> 9 [ style=dashed ]
      9 -> 11 [ ]
      11 -> 12 [ style=dashed ]
      12 -> 13 [ ]
      13 -> 14 [ ]
      14 -> 16 [ ]
    }
    ```
    
    To convert the DOT description into an image. A quick way is to use online visualization
tools (e.g. http://www.webgraphviz.com, just copy and paste the graph description into the
webpage's textbox and click _Generate Graph!_.
    
    The output can also be redirected to a file and then converted to an image with the dot
tool, e.g.
    ```
    quickstep_cli_shell -visualize_execution_dag=true < some_query.sql 2> some_graph.dot
    
    dot -Gsize="8,10.5" -Teps some_graph.dot -o some_graph.eps
    ```
    Here eps can be `pdf`, `png`, etc.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/incubator-quickstep execution-dag-visualizer

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-quickstep/pull/76.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #76
    
----
commit 4b944328afde90c961122d547c6fa4a8d0230c1c
Author: Jianqiao Zhu <jianqiao@cs.wisc.edu>
Date:   2016-08-02T21:57:47Z

    Add visualization for execution plan DAGs combined with profiling stats

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message