Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 7BA83200CE4 for ; Sun, 20 Aug 2017 10:41:00 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 7A29F164909; Sun, 20 Aug 2017 08:41:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 720D51648FD for ; Sun, 20 Aug 2017 10:40:59 +0200 (CEST) Received: (qmail 33102 invoked by uid 500); 20 Aug 2017 08:40:57 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 33092 invoked by uid 99); 20 Aug 2017 08:40:56 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Aug 2017 08:40:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 8633BC1D8F for ; Sun, 20 Aug 2017 08:40:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.151 X-Spam-Level: X-Spam-Status: No, score=-0.151 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Dl666oXpc4qw for ; Sun, 20 Aug 2017 08:40:54 +0000 (UTC) Received: from mail-qk0-f175.google.com (mail-qk0-f175.google.com [209.85.220.175]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 6B0D360D99 for ; Sun, 20 Aug 2017 08:40:54 +0000 (UTC) Received: by mail-qk0-f175.google.com with SMTP id x77so26937498qka.5 for ; Sun, 20 Aug 2017 01:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=YW8YjKD9sSOpCdloY5uRLj2zxuHd7iIcCa0SucemVpE=; b=ollXdm8GPoL/sXU/52WsaFlZte1Yt7P5AfYyR/I2iIXRqI6XwK/uye3nDkAQHsOqD8 wfqq5dFIlXss0nVnPaVscV0Apo3NnjAKBhbQB6Y0mFgd00HXatV36g+t0u//9z+1St1P qqqeRdw6BgCvrdI+8Zse/DiF7VGFj/KGDfc2LyEQx26Y//xmBQRzEr7FWBdTUb+Gx03v 7ALBmahE5WIUFxU6vATS0tpjPTEn81S0QkB5PqenXll8fS/L7ceNw4NDA0kn76OOL8yr maGw2T/8yK18zNfb8CDqGD2BJ32t5zFkReTwmLe7fHkrCzoV50ahKjRxr1kXBNm72uYl aWxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=YW8YjKD9sSOpCdloY5uRLj2zxuHd7iIcCa0SucemVpE=; b=TRyqV5HxymbwjIo03cGzOdFtpX2gGgMuKYEtFPOmf2g3a+mqrpnsW6YMmEDJcC6C+L gjabbL+fNnKRgJZhENId3DRtlpoF6TXLIk5jA4zPdevf6pa3gY6sGBfS1QVVMcVau4kM ndMHDz0Zp83oMWHxteaypMvYkYqVv2c9gQb6+M3Y/SzqMda0MzF01gMhkpw4BQIg2UeD BikBH2xjWP+MvjESJJQcz0LcF1x21NdcZPRd5DRjGapNHXXXiNKtFqF6qQ0IyvxyaJoZ KArqauzMsnOtqO7aMzIaTjk4KLFhJ/8XyMVFTa+bWgm5K7HKQueWRbZGAYMI9M4zej+2 cMJA== X-Gm-Message-State: AHYfb5gJBjWFjyJzc8kKvuBs9NGGUevyJNspNDrCOPvvljL5BnZR+M2l nowQYJbopA1lf5B3YdyD53oOzufbxg== X-Received: by 10.55.149.7 with SMTP id x7mr17937128qkd.14.1503218454029; Sun, 20 Aug 2017 01:40:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.12.128.170 with HTTP; Sun, 20 Aug 2017 01:40:13 -0700 (PDT) In-Reply-To: <6D51C89C-FD56-49F6-88B2-B9F37BA0DF17@comcast.net> References: <965AE5F4-8F70-46BA-B529-FB49660769C5@gmail.com> <6D51C89C-FD56-49F6-88B2-B9F37BA0DF17@comcast.net> From: kant kodali Date: Sun, 20 Aug 2017 01:40:13 -0700 Message-ID: Subject: Re: Does Spark SQL uses Calcite? To: Jules Damji Cc: Russell Spitzer , Ryan , Sathish Kumaran Vairavelu , "user @spark" Content-Type: multipart/alternative; boundary="94eb2c083192f8ec8f05572b51b3" archived-at: Sun, 20 Aug 2017 08:41:00 -0000 --94eb2c083192f8ec8f05572b51b3 Content-Type: text/plain; charset="UTF-8" Hi Jules, I am looking to connect to Spark via JDBC so I can run Spark SQL queries via JDBC but not use SPARK SQL to connect to other JDBC sources. Thanks! On Sat, Aug 19, 2017 at 5:54 PM, Jules Damji wrote: > Try this link to see how you may connect https://docs.databricks.com/ > spark/latest/data-sources/sql-databases.html > > Cheers > Jules > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Aug 19, 2017, at 5:27 PM, kant kodali wrote: > > Hi Russell, > > I went through this https://jaceklaskowski.gitbooks.io/mastering-apache- > spark/spark-sql-thrift-server.html and I am still a bit confused on what > hive is doing in here ? Is there any example I can look at on how to talk > to Spark using Spark SQL JDBC driver alone and not hive ? > > Thanks, > kant > > > > > > On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer < > russell.spitzer@gmail.com> wrote: > >> You don't have to go through hive. It's just spark sql. The application >> is just a forked hive thrift server. >> >> On Fri, Aug 11, 2017 at 8:53 PM kant kodali wrote: >> >>> @Ryan it looks like if I enable thrift server I need to go through hive. >>> I was talking more about having JDBC connector for Spark SQL itself other >>> words not going through hive. >>> >>> On Fri, Aug 11, 2017 at 6:50 PM, kant kodali wrote: >>> >>>> @Ryan Does it work with Spark SQL 2.1.1? >>>> >>>> On Fri, Aug 11, 2017 at 12:53 AM, Ryan wrote: >>>> >>>>> the thrift server is a jdbc server, Kanth >>>>> >>>>> On Fri, Aug 11, 2017 at 2:51 PM, wrote: >>>>> >>>>>> I also wonder why there isn't a jdbc connector for spark sql? >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> On Aug 10, 2017, at 2:45 PM, Jules Damji wrote: >>>>>> >>>>>> Yes, it's more used in Hive than Spark >>>>>> >>>>>> Sent from my iPhone >>>>>> Pardon the dumb thumb typos :) >>>>>> >>>>>> On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu < >>>>>> vsathishkumaran@gmail.com> wrote: >>>>>> >>>>>> I think it is for hive dependency. >>>>>> On Thu, Aug 10, 2017 at 4:14 PM kant kodali >>>>>> wrote: >>>>>> >>>>>>> Since I see a calcite dependency in Spark I wonder where Calcite is >>>>>>> being used? >>>>>>> >>>>>>> On Thu, Aug 10, 2017 at 1:30 PM, Sathish Kumaran Vairavelu < >>>>>>> vsathishkumaran@gmail.com> wrote: >>>>>>> >>>>>>>> Spark SQL doesn't use Calcite >>>>>>>> >>>>>>>> On Thu, Aug 10, 2017 at 3:14 PM kant kodali >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> Does Spark SQL uses Calcite? If so, what for? I thought the Spark >>>>>>>>> SQL has catalyst which would generate its own logical plans, physical plans >>>>>>>>> and other optimizations. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Kant >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>> > --94eb2c083192f8ec8f05572b51b3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Jules,

I am looking to co= nnect to Spark via JDBC so I can run Spark SQL queries via JDBC but not use= SPARK SQL to connect to other JDBC sources.

Thank= s!

On = Sat, Aug 19, 2017 at 5:54 PM, Jules Damji <dmatrix@comcast.net> wrote:

<= div id=3D"m_8433611457728324769AppleMailSignature">Cheers
Jules

Sent from my iPhon= e
Pardon the dumb thumb typos :)

On Aug 19, 2017, at 5:27 PM, kant kodali <kanth909@gmail.com> wrote:
=
Hi Russell,
<= br>
I went through this=C2=A0https://jaceklaskowski.gitbooks.io/mastering-apache-spark/s= park-sql-thrift-server.html and I am still a bit confused on what = hive is doing in here ? Is there any example I can look at on how to talk t= o Spark using Spark SQL JDBC driver alone and not hive ?

Thanks,
kant


=


On Sat, Aug 12, 2017 at 6:07 AM, Russell Spitzer <russell.spitzer@gmail.com> wrote:
You don't have to go through hive. It's= just spark sql. The application is just a forked hive thrift server.
=

On Fri, Aug 11, 2017= at 8:53 PM kant kodali <kanth909@gmail.com> wrote:
@Ryan it looks like if I enable thrift server I ne= ed to go through hive. I was talking more about having JDBC connector for S= park SQL itself other words not going through hive.

On Fri, Aug 11, 2017 at 6:50 PM, ka= nt kodali <kanth909@gmail.com> wrote:
@Ryan Does it work with Spark SQL 2.1.1?

On Fri, Aug 11, 2017 at 12:53 AM, Ryan <ryan.hd.ren@gmail.com&g= t; wrote:
the thr= ift server is a jdbc server, Kanth

On Fri, Aug 11, 2017 at 2:51 PM, <kanth909@gmail.com> wrote:
I also = wonder why there isn't a jdbc connector for spark sql?

Sent from= my iPhone
=
On Aug 10, 2017, at 2:45 PM, Jules Damji <dmatrix@comcast.net> wrote:

Yes, it's more used in Hive tha= n Spark=C2=A0

Sent from my iPhone
Pardon the dumb thumb typos :)=

On Aug 10, 2017, at 2:24 PM, Sathish Kumaran Vairavelu= <vsathis= hkumaran@gmail.com> wrote:

I think it is for hive dependency.
On Thu, Aug 10, 2017 at 4:14 PM kant kodali <kanth909@gmail.com> wrote:
Since I see a calcit= e dependency in Spark I wonder where Calcite is being used?

On Thu, Aug 10, 2017 at 1:3= 0 PM, Sathish Kumaran Vairavelu <vsathishkumaran@gmail.com>= wrote:
Spark SQL doesn't use = Calcite

On Thu, Aug 10, 2017 at 3:14 PM kant kodali <kanth909@gmail.com>= wrote:
Hi All,=C2= =A0

Does Spark SQL uses Calcite? If so, what for? I thou= ght the Spark SQL has catalyst which would generate its own logical plans, = physical plans and other optimizations.=C2=A0

= Thanks,
Kant

=




--94eb2c083192f8ec8f05572b51b3--