crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-438) Visualizations of some important internal/intermediate pipeline planning states
Date Fri, 05 Dec 2014 00:07:13 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234821#comment-14234821
] 

Josh Wills commented on CRUNCH-438:
-----------------------------------

I think it has functional value-- my customers are always happy to have more tools for debugging
pipelines, so I don't mind the additional hooks into the planner code. My one concern would
be around making the inner planner classes like Vertex and NodePath accessible from outside
of the plan package vs. keeping all of the (extensive) dotfile code underneath the plan package,
but I could be convinced either way on that one.

For the time being, I didn't have any plans to add lifecycle hooks into the planner for other
use cases-- what are some examples of things that people would want to do there?

> Visualizations of some important internal/intermediate pipeline planning states
> -------------------------------------------------------------------------------
>
>                 Key: CRUNCH-438
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-438
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.10.0, 0.8.3
>            Reporter: Christian Tzolov
>            Assignee: Christian Tzolov
>         Attachments: CRUNCH-438.2.patch, CRUNCH-438.3.patch, CRUNCH-438.4.patch, CRUNCH-438.5.patch,
CRUNCH-438.patch
>
>
> To improve the understability of the pipeline planning stages it would help to visualize
some intermediate planning states like:
> - PCollection lineage. (visualizing the output-pcollection-targets structure) 
> - MSCRPlanner's planning Graphs before and after the split up of dependent GBK nodes
> - RTNode hierarchy along with the Input and Output configurations as persistent in the
Configuration before the execution of the pipeline. 
> Most of the information can be intercepted in the MSCRPlanner#plan()  method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message