spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng, Hao" <hao.ch...@intel.com>
Subject RE: Join Order Optimization
Date Mon, 12 Oct 2015 04:51:20 GMT
Probably you have to read the source code, I am not sure if there are any .ppt or slides.

Hao

From: VJ Anand [mailto:vjanand@sankia.com]
Sent: Monday, October 12, 2015 11:43 AM
To: Cheng, Hao
Cc: Raajay; user@spark.apache.org
Subject: Re: Join Order Optimization

Hi - Is there a design document for those operations that have been implemented in 1.4.0?
if so,where can I find them
-VJ

On Sun, Oct 11, 2015 at 7:27 PM, Cheng, Hao <hao.cheng@intel.com<mailto:hao.cheng@intel.com>>
wrote:
Yes, I think the SPARK-2211 should be the right place to follow the CBO stuff, but probably
that will not happen right away.

The jira issue introduce the statistic info can be found at:
https://issues.apache.org/jira/browse/SPARK-2393

Hao

From: Raajay [mailto:raajay.v@gmail.com<mailto:raajay.v@gmail.com>]
Sent: Monday, October 12, 2015 10:17 AM
To: Cheng, Hao
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Join Order Optimization

Hi Cheng,
Could you point me to the JIRA that introduced this change ?

Also, is this SPARK-2211 the right issue to follow for cost-based optimization?
Thanks
Raajay


On Sun, Oct 11, 2015 at 7:57 PM, Cheng, Hao <hao.cheng@intel.com<mailto:hao.cheng@intel.com>>
wrote:
Spark SQL supports very basic join reordering optimization, based on the raw table data size,
this was added couple major releases back.

And the “EXPLAIN EXTENDED query” command is a very informative tool to verify whether
the optimization taking effect.

From: Raajay [mailto:raajay.v@gmail.com<mailto:raajay.v@gmail.com>]
Sent: Sunday, October 11, 2015 9:22 AM
To: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Join Order Optimization

Hello,
Does Spark-SQL support join order optimization as of the 1.5.1 release ? From the release
notes, I did not see support for this feature, but figured will ask the users-list to be sure.
Thanks
Raajay




--
VJ Anand
Founder
Sankia
vjanand@sankia.com<mailto:vjanand@sankia.com>
925-640-1340
www.sankia.com<http://www.sankia.com>

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use
of the intended recipient(s) and may contain confidential and privileged information. Any
unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all copies of the original
message
Mime
View raw message