crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Tzolov (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-438) Visualizations of some important internal/intermediate pipeline planning states
Date Mon, 07 Jul 2014 11:05:34 GMT


Christian Tzolov updated CRUNCH-438:

    Attachment: CRUNCH-438.2.patch

Updated patch that writes the dotfile content in the Configuration using the following placeholders:
PlanningParameters.PIPELINE_PLAN_DOTFILE;   (original one)

One can print a dotfile content like this:

For the experiment i've also  integrate this with the PlanningParameters.PIPELINE_DOTFILE_OUTPUT_DIR
(CRUNCH-418) If the PIPELINE_DOTFILE_OUTPUT_DIR path is set then 5 dotfiles will be produced.

I agree with Gabriel Reid that those diagrams are more like a debug tool. I the PIPELINE_DOTFILE_OUTPUT_DIR
is not for debugging purpose? then  perhaps I should revert this integration?

- I've fixed the RTNode#getEmitter() method name. The reason it isn't called is that the emitters
are created during the configuration stage. So this field is empty during the planning stage.
Maybe we can find a use of it if we decide to create a live/run-time diagram representation

> Visualizations of some important internal/intermediate pipeline planning states
> -------------------------------------------------------------------------------
>                 Key: CRUNCH-438
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.10.0, 0.8.3
>            Reporter: Christian Tzolov
>            Assignee: Christian Tzolov
>         Attachments: CRUNCH-438.2.patch, CRUNCH-438.patch
> To improve the understability of the pipeline planning stages it would help to visualize
some intermediate planning states like:
> - PCollection lineage. (visualizing the output-pcollection-targets structure) 
> - MSCRPlanner's planning Graphs before and after the split up of dependent GBK nodes
> - RTNode hierarchy along with the Input and Output configurations as persistent in the
Configuration before the execution of the pipeline. 
> Most of the information can be intercepted in the MSCRPlanner#plan()  method.

This message was sent by Atlassian JIRA

View raw message