spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Murphy <halcyo...@gmail.com>
Subject Re: query planner design doc?
Date Fri, 23 Jan 2015 17:44:28 GMT
Okay, thanks.  The design document mostly details the infrastructure for optimization strategies
but doesn’t detail the strategies themselves.  I take it the set of strategies are basically
embodied in SparkStrategies.scala...is there a design doc/roadmap/JIRA issue detailing what
strategies exist and which are planned?

Thanks,
Nick

> On Jan 22, 2015, at 7:45 PM, Michael Armbrust <michael@databricks.com> wrote:
> 
> Here is the initial design document for catalyst :
> https://docs.google.com/document/d/1Hc_Ehtr0G8SQUg69cmViZsMi55_Kf3tISD9GPGU5M1Y/edit
<https://docs.google.com/document/d/1Hc_Ehtr0G8SQUg69cmViZsMi55_Kf3tISD9GPGU5M1Y/edit>
> 
> Strategies (many of which are in SparkStragegies.scala) are the part that creates the
physical operators from a catalyst logical plan.  These operators have execute() methods that
actually call RDD operations.
> 
> On Thu, Jan 22, 2015 at 3:19 PM, Nicholas Murphy <halcyonic@gmail.com <mailto:halcyonic@gmail.com>>
wrote:
> Hi-
> 
> Quick question: is there a design doc (or something more than “look at the code”)
for the query planner for Spark SQL (i.e., the component that takes…Catalyst?…operator
trees and translates them into SPARK operations)?
> 
> Thanks,
> Nick
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org <mailto:dev-unsubscribe@spark.apache.org>
> For additional commands, e-mail: dev-help@spark.apache.org <mailto:dev-help@spark.apache.org>
> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message