crunch-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject svn commit: r1650710 - /crunch/site/trunk/content/user-guide.mdtext
Date Sat, 10 Jan 2015 01:40:26 GMT
Author: tzolov
Date: Sat Jan 10 01:40:25 2015
New Revision: 1650710

CRUNCH-487: Document the pipeline execution plan visualization configuration.	


Modified: crunch/site/trunk/content/user-guide.mdtext
--- crunch/site/trunk/content/user-guide.mdtext (original)
+++ crunch/site/trunk/content/user-guide.mdtext Sat Jan 10 01:40:25 2015
@@ -1602,3 +1602,20 @@ By embedding our business logic inside o
 computations that combine custom DoFns with Crunch's built-in `cogroup` operation by using
the [MemPipeline](#mempipeline)
 implementation to create test data sets that we can easily verify by hand, and then this
same logic can be executed on
 a distributed data set using either the [MRPipeline](#mrpipeline) or [SparkPipeline](#sparkpipeline)
+### Pipeline execution plan visualizations
+Crunch provides tools to visualize the pipeline execution plans. The [PipelineExecution](apidocs/0.10.0/org/apache/crunch/PipelineExecution.html)
+`String getPlanDotFile()` method returns an execution plan visualization in DOT format. If
the dot file output folder property is set, Crunch produces a DOT file after each pipeline
+Additional aspects of the execution plans are provided when the DOT file debug mode is enabled.
Then Crunch provides 4 additional DOT diagrams visualizing different internal stages of the
execution plan. Such plans include PCollection lineage, Base graph plan, Split graph plans,
Run-time nodes. 
+Note: To enable the debug mode you should set an out put folder first. The following snapped
switches the DOT file debug mode.  As a result 5 DOT diagrams are generated in the output
folder after each Pipeline execution:
+	    Configuration conf = ...
+	    String dotfileDir = ...
+	    DotfileUtills.setPipelineDotfileOutputDir(conf, dotfileDir);
+	    DotfileUtills.enableDebugDotfiles(conf);

View raw message