Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E8AEE18986 for ; Mon, 12 Oct 2015 03:43:11 +0000 (UTC) Received: (qmail 24003 invoked by uid 500); 12 Oct 2015 03:43:07 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 23897 invoked by uid 500); 12 Oct 2015 03:43:06 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 23887 invoked by uid 99); 12 Oct 2015 03:43:06 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2015 03:43:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 32547C56D2 for ; Mon, 12 Oct 2015 03:43:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.001 X-Spam-Level: **** X-Spam-Status: No, score=4.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id eLFhxmaNAWDF for ; Mon, 12 Oct 2015 03:42:57 +0000 (UTC) Received: from mail-vk0-f51.google.com (mail-vk0-f51.google.com [209.85.213.51]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 0102D207EE for ; Mon, 12 Oct 2015 03:42:56 +0000 (UTC) Received: by vkaw128 with SMTP id w128so23229251vka.0 for ; Sun, 11 Oct 2015 20:42:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=hIxhf0c/nCg1u0PehHT0TyxNEeXjtd0YuVzxwMwUgNY=; b=VeAJQHJkui3fFBmt0ESkLVEz9l6fz7Zod2KNlnDiTzEvGOPH1KLZ168YuDKubSrUJY MYl2Fxqn/Y3gu+NB75FQsRKI65tj7/uGjGBcIciaND8oe6Vrhfl3GwT3vynGd0J1Gp+u sNObkTicqzNNUu37XHDPG6yzgC/PFNdfMak/9kn1L9Ekeo2oe4s3C31oYoAdY1jX+Dpg ZuISaeD/VRcOzWKp+5YaHAbUE+vWH/uftALgEL5e7hecy3oolDUkCWRPXflCiQN83dQD FOtR+A0hNbrb8M0MQUgm0OEALcvFET4iqgUPJdbT9TvkvxW/XGp/s2R1ugEharEqb8U4 nDLQ== X-Gm-Message-State: ALoCoQlB3hmD6fOm44wgsMykEujlDgk477FRchnZ9yYY6GPl2xDgguBkCAJT9pYzKjrFeIZGQKaK MIME-Version: 1.0 X-Received: by 10.31.180.1 with SMTP id d1mr2783542vkf.131.1444621375867; Sun, 11 Oct 2015 20:42:55 -0700 (PDT) Received: by 10.31.51.200 with HTTP; Sun, 11 Oct 2015 20:42:55 -0700 (PDT) X-Originating-IP: [73.223.141.237] In-Reply-To: <80833ADD533E324CA05C160E41B6366103CF2689@shsmsx102.ccr.corp.intel.com> References: <80833ADD533E324CA05C160E41B6366103CF260E@shsmsx102.ccr.corp.intel.com> <80833ADD533E324CA05C160E41B6366103CF2689@shsmsx102.ccr.corp.intel.com> Date: Sun, 11 Oct 2015 20:42:55 -0700 Message-ID: Subject: Re: Join Order Optimization From: VJ Anand To: "Cheng, Hao" Cc: Raajay , "user@spark.apache.org" Content-Type: multipart/alternative; boundary=001a11438b92f1a8fd0521e01f7e --001a11438b92f1a8fd0521e01f7e Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi - Is there a design document for those operations that have been implemented in 1.4.0? if so,where can I find them -VJ On Sun, Oct 11, 2015 at 7:27 PM, Cheng, Hao wrote: > Yes, I think the SPARK-2211 should be the right place to follow the CBO > stuff, but probably that will not happen right away. > > > > The jira issue introduce the statistic info can be found at: > > https://issues.apache.org/jira/browse/SPARK-2393 > > > > Hao > > > > *From:* Raajay [mailto:raajay.v@gmail.com] > *Sent:* Monday, October 12, 2015 10:17 AM > *To:* Cheng, Hao > *Cc:* user@spark.apache.org > *Subject:* Re: Join Order Optimization > > > > Hi Cheng, > > Could you point me to the JIRA that introduced this change ? > > > Also, is this SPARK-2211 the right issue to follow for cost-based > optimization? > > Thanks > > Raajay > > > > > > On Sun, Oct 11, 2015 at 7:57 PM, Cheng, Hao wrote: > > Spark SQL supports very basic join reordering optimization, based on the > raw table data size, this was added couple major releases back. > > > > And the =E2=80=9CEXPLAIN EXTENDED query=E2=80=9D command is a very inform= ative tool to > verify whether the optimization taking effect. > > > > *From:* Raajay [mailto:raajay.v@gmail.com] > *Sent:* Sunday, October 11, 2015 9:22 AM > *To:* user@spark.apache.org > *Subject:* Join Order Optimization > > > > Hello, > > Does Spark-SQL support join order optimization as of the 1.5.1 release ? > From the release notes, I did not see support for this feature, but figur= ed > will ask the users-list to be sure. > > Thanks > > Raajay > > > --=20 *VJ Anand* *Founder * *Sankia* vjanand@sankia.com 925-640-1340 www.sankia.com *Confidentiality Notice*: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message --001a11438b92f1a8fd0521e01f7e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi - Is there a design document for those operations that = have been implemented in 1.4.0? if so,where can I find them
-VJ

On Sun, Oct 1= 1, 2015 at 7:27 PM, Cheng, Hao <hao.cheng@intel.com> wrote= :

Yes, I think the SPARK-2211 should be= the right place to follow the CBO stuff, but probably that will not happen= right away.

=C2=A0

The jira issue introduce the statisti= c info can be found at:

https://issues.apache.org/jira/br= owse/SPARK-2393

=C2=A0

Hao

=C2=A0

From: Raajay [mailto:raajay.v@gmail.com]
Sent: Monday, October 12, 2015 10:17 AM
To: Cheng, Hao
Cc: user@= spark.apache.org
Subject: Re: Join Order Optimization

=C2=A0

Hi Cheng,

Could you point me to the JIRA that introduced this = change ?


Also, is this SPARK-2211 the right issue to follow for cost-based optimizat= ion?

Thanks

Raajay

=C2=A0<= /p>

=C2=A0

On Sun, Oct 11, 2015 at 7:57 PM, Cheng, Hao <hao.cheng@intel.com> wrote:

Spark SQL supports very basic join re= ordering optimization, based on the raw table data size, this was added couple major releases back.

=C2=A0

And the =E2=80=9CEXPLAIN EXTENDED que= ry=E2=80=9D command is a very informative tool to verify whether the optimi= zation taking effect.

=C2=A0

From: Raajay [mailto:raajay.v@gmail.com]
Sent: Sunday, October 11, 2015 9:22 AM
To: user@= spark.apache.org
Subject: Join Order Optimization

=C2=A0

Hello,<= /p>

Does Spark-SQL suppor= t join order optimization as of the 1.5.1 release ? From the release notes,= I did not see support for this feature, but figured will ask the users-lis= t to be sure.

Thanks

Raajay

=C2=A0




--
VJ Anand
Founder= =C2=A0
Sankia=C2= =A0
925-640-1340

Confidentiality Notice: This e-mail message, including any attachments, is for the sole= use of the intended recipient(s) and may contain confidential and privileg= ed information. Any unauthorized review, use, disclosure or distribution is= prohibited. If you are not the intended recipient, please contact the send= er by reply e-mail and destroy all copies of the original message
--001a11438b92f1a8fd0521e01f7e--