crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul Sharma <rahul0...@gmail.com>
Subject Re: Visualize DAG of a pipeline
Date Tue, 05 Feb 2013 03:32:36 GMT
Yes, a dot language file is generated in the pipeline. The file is a
visualization of how MR jobs have been executed in the pipeline. You can
access the same like :

String dotFileContents =
pipeline.getConfiguration().get(PlanningParameters.PIPELINE_PLAN_DOTFILE);

The file can be analyzed with various tools like Graphviz. For more on DOT
please check http://en.wikipedia.org/wiki/DOT_language


On Tue, Feb 5, 2013 at 8:49 AM, Josh Wills <jwills@cloudera.com> wrote:

> +greid
>
> Gabriel wrote one, IIRC-- I think that a .dot file with the plan for the
> job gets embedded in the Configuration object returned from the planner.
>
>
> On Mon, Feb 4, 2013 at 7:13 PM, Chao Shi <stepinto@live.com> wrote:
>
>> Hi crunch users,
>>
>> I would like to know if there are any tool to help me understand crunch
>> optimized MR stages.
>>
>> Particularly, I think I need to see the DAG of job stages. I'm writing a
>> pipeline consists of several joins. The pipeline produces significant
>> more intermediate output than I expect. I want to investigate what's going
>> wrong there.
>>
>> Thanks,
>> Chao
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>

Mime
View raw message