incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-105) Add generation of Graphviz dot files to describe the pipeline job plan
Date Wed, 31 Oct 2012 07:53:13 GMT


Gabriel Reid commented on CRUNCH-105:

Thanks for the feedback! Moving the config key to PlanningParameters sounds like a good plan.
Putting it in MSCRPlanner didn't feel very right to me, but I didn't have a better idea of
where to put it at the time.

The dotfile info is indeed available on a pipeline that's been executed as well -- the internal
call to the planner populates it whether you run the pipeline or not.

I'd like to refactor the code a bit, but that will also involve refactoring the some other
things in the planner to make the internal information more accessible (the dotfile writer
has a bit more logic in it than what I would have liked right now). I see this as a first
step to documenting and refactoring the planner (as it makes it easier to understand what's
going on), but there's still work that can be done here.

I'd also like to add dotfiles for the two stages of the pre-planning process (initial graph
and graph with intermediate files), as well as making this more accessible via the API.

I think that this is a useful update for the 0.4.0 release (and I'm going to be offline for
a few days), so I'll move the config key and commit it, and then tackle the planner refactoring,
pre-planning graph building, and API accessibility of the dotfiles separately.
> Add generation of Graphviz dot files to describe the pipeline job plan
> ----------------------------------------------------------------------
>                 Key: CRUNCH-105
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.4.0
>            Reporter: Gabriel Reid
>            Assignee: Gabriel Reid
>         Attachments: CRUNCH-105.patch, innerjoingraph.png
> It could be very useful to have the ability to create Graphviz[1] dot files[2] based
on the outcome of the planner. This could facilitate visualizing the full graph of DoFns as
well as jobs and mappers and reducers in a Crunch pipeline.
> [1]
> [2] 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message