crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Tzolov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-438) Visualizations of some important internal/intermediate pipeline planning states
Date Sun, 06 Jul 2014 18:57:33 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053182#comment-14053182
] 

Christian Tzolov commented on CRUNCH-438:
-----------------------------------------

I wanted to discuss what is the best way to integrate it. Maybe the fastest way is to extend
the PlanningParameters.PIPELINE_DOTFILE_OUTPUT_DIR  approach and generate few extra files
in the same location.  
I have been thinking also of abstracting a callback (observer), interceptor interface implemented
by the MSCRPlanner. The reason is to troy to separate the tracing code from the main code
as we can have more diagrams and different way to store/publish them.  But this might be worth
a separate issue when needed for now i will focus on the PIPELINE_DOTFILE_OUTPUT_DIR approach


> Visualizations of some important internal/intermediate pipeline planning states
> -------------------------------------------------------------------------------
>
>                 Key: CRUNCH-438
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-438
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.10.0, 0.8.3
>            Reporter: Christian Tzolov
>            Assignee: Christian Tzolov
>         Attachments: CRUNCH-438.patch
>
>
> To improve the understability of the pipeline planning stages it would help to visualize
some intermediate planning states like:
> - PCollection lineage. (visualizing the output-pcollection-targets structure) 
> - MSCRPlanner's planning Graphs before and after the split up of dependent GBK nodes
> - RTNode hierarchy along with the Input and Output configurations as persistent in the
Configuration before the execution of the pipeline. 
> Most of the information can be intercepted in the MSCRPlanner#plan()  method.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message