Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 94EA119C4C for ; Wed, 13 Apr 2016 17:44:35 +0000 (UTC) Received: (qmail 66350 invoked by uid 500); 13 Apr 2016 17:44:32 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 66240 invoked by uid 500); 13 Apr 2016 17:44:32 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 66227 invoked by uid 99); 13 Apr 2016 17:44:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Apr 2016 17:44:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 9D16DC36E3 for ; Wed, 13 Apr 2016 17:44:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id bMKS9q-eHI7F for ; Wed, 13 Apr 2016 17:44:29 +0000 (UTC) Received: from mail-vk0-f66.google.com (mail-vk0-f66.google.com [209.85.213.66]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id B90E15F3F4 for ; Wed, 13 Apr 2016 17:44:28 +0000 (UTC) Received: by mail-vk0-f66.google.com with SMTP id e185so7345687vkb.2 for ; Wed, 13 Apr 2016 10:44:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=gDSTLxjpmThoxprPPf41JhAdCR5EIFF4N0H1/rfUlzE=; b=TVKEi0ixJ3yeI3okc1OfY8byMWRs03FZaF4mC4DStT49EpNPCaxR6RJJxcpANzfvXb S8/3GIsv5+vSGhvWpG/g2cGPmI12ZobxOUnCwjKLb4+aM31ROr/+9Ik4eKE58aHvcxs/ eS9LP1y9TmiQTyqfT8iRllZFCMvFAQFvQRNbinKN5wjk2+N5/+0JBBICaT9HuW0AwyRW iFHPjvPOZGU0pEwRH/iAp8Bi1zJar65OIzMKp8RBx4mH7AEaKAAHLIA6OowYDz7KX0gc SKA1CLo1KpclDB4GgKSfuFkJ9nFPGtNNag+vxCJhsjjEJQjNsdmiD6zeaIwiPy9YAfmS QFYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=gDSTLxjpmThoxprPPf41JhAdCR5EIFF4N0H1/rfUlzE=; b=eitbswM6GxAwAS3sxQBd0JnrQesb7d7JrzBCY5RxLq+PFWpB9ffDjGJ2laZHW+u4th kEztYm4mKRl5fgxfv3HRIBuzqxeckINC7DpOCry90cMxMPVS1YHsE/tqsydfSTRZLLiF PvD5D7embH9XK+bNMW5JE0zY6UUlu/VTZvVSRuZnU0GOdrs0trTzuk52E8yJ3mp0MQYS bOZWz9xikuffuJadlMauu2Z4l9zaaJef/rMWasU9jPqViV3J3DJhOfQVNDf9D1+ARHqS NM/eSLeYkUmiZUt4DGkn+q7QW2rLq4DbQw4rOavgMHjhE9X0wBjrMwWSQbkYuMf1gcZx L88g== X-Gm-Message-State: AOPr4FVy18lCI+8csiF2Zqlng0o/dVNBIoLijDOshxf4pB4p83Y6U2xj9UqfmFuCzHm0stp+6F8Gs1V8Ruz+6g== MIME-Version: 1.0 X-Received: by 10.176.6.4 with SMTP id f4mr5202555uaf.95.1460569467332; Wed, 13 Apr 2016 10:44:27 -0700 (PDT) Received: by 10.31.229.129 with HTTP; Wed, 13 Apr 2016 10:44:27 -0700 (PDT) In-Reply-To: References: Date: Wed, 13 Apr 2016 18:44:27 +0100 Message-ID: Subject: Re: Silly question... From: Mich Talebzadeh To: Michael Segel Cc: "user @spark" Content-Type: multipart/alternative; boundary=94eb2c048cb4453c67053061545d --94eb2c048cb4453c67053061545d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable These are the components *java -versionjava version "1.8.0_77"*Java(TM) SE Runtime Environment (build 1.8.0_77-b03) Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode) *hadoop versionHadoop 2.6.0*Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1 Compiled by jenkins on 2014-11-13T21:10Z Compiled with protoc 2.5.0 *hive --versionHive 2.0.0* ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ *version 1.6.1* /_/ *Using Scala version 2.10.5 (*Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77) Metastore BANNER CON_ID ---------------------------------------------------------------------------= ----- ---------- *Oracle Database 12c* Enterprise Edition Release 12.1.0.2.0 - 64bit Production To me all working OK HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxianrbJd6= zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 13 April 2016 at 18:10, Michael Segel wrote: > Mich > > Are you building your own releases from the source? > Which version of Scala? > > Again, the builds seem to be ok and working, but I don=E2=80=99t want to = hit some > =E2=80=98gotcha=E2=80=99 if I could avoid it. > > > On Apr 13, 2016, at 7:15 AM, Mich Talebzadeh > wrote: > > Hi, > > I am not sure this helps. > > we use Spark 1.6 and Hive 2. I also use JDBC (beeline for Hive) plus > Oracle and Sybase. They all work fine. > > > HTH > > Dr Mich Talebzadeh > > > LinkedIn * https://www.linkedin.com/profile/view?id=3DAAEAAAAWh2gBxianrbJ= d6zP6AcPCCdOABUrV8Pw > * > > > http://talebzadehmich.wordpress.com > > > > On 12 April 2016 at 23:42, Michael Segel > wrote: > >> Hi, >> This is probably a silly question on my part=E2=80=A6 >> >> I=E2=80=99m looking at the latest (spark 1.6.1 release) and would like t= o do a >> build w Hive and JDBC support. >> >> From the documentation, I see two things that make me scratch my head. >> >> 1) Scala 2.11 >> "Spark does not yet support its JDBC component for Scala 2.11.=E2=80=9D >> >> So if we want to use JDBC, don=E2=80=99t use Scala 2.11.x (in this case = its >> 2.11.8) >> >> 2) Hive Support >> "To enable Hive integration for Spark SQL along with its JDBC server and >> CLI, add the -Phive and Phive-thriftserver profiles to your existing >> build options. By default Spark will build with Hive 0.13.1 bindings.=E2= =80=9D >> >> So if we=E2=80=99re looking at a later release of Hive=E2=80=A6 lets say= 1.1.x =E2=80=A6 still >> use the -Phive and Phive-thriftserver . Is there anything else we should >> consider? >> >> Just asking because I=E2=80=99ve noticed that this part of the documenta= tion >> hasn=E2=80=99t changed much over the past releases. >> >> Thanks in Advance, >> >> -Mike >> >> > > --94eb2c048cb4453c67053061545d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
These are the components

java -version
java v= ersion "1.8.0_77"
Java(TM) SE Runtime Environment (bu= ild 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mi= xed mode)

hadoop version
Hadoop 2.6.0
Subversion https://git-wip-us.= apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e3= 3bb1
Compiled by jenkins on 2014-11-13T21:10Z
Compiled with protoc 2.= 5.0

hive --version
Hive 2.0.0
<= /div>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ____=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __
=C2=A0=C2=A0=C2=A0=C2=A0 /= __/__=C2=A0 ___ _____/ /__
=C2=A0=C2=A0=C2=A0 _\ \/ _ \/ _ `/ __/=C2=A0= '_/
=C2=A0=C2=A0 /___/ .__/\_,_/_/ /_/\_\=C2=A0=C2=A0 versi= on 1.6.1
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /_/
Using Scala vers= ion 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77)
<= /font>

Metastore

=C2=A0BANNER=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= CON_ID
----------------------------------------------------------------= ---------------- ----------
Oracle Database 12c Enterpr= ise Edition Release 12.1.0.2.0 - 64bit Production=C2=A0=C2=A0
=

To me all working OK

HTH




On 13 April 2016 at 18:10, Michael Segel <msegel_hadoop@hotmail.com> wrote:
Mich

=
Are you building your own releases from the source?=C2=A0
Wh= ich version of Scala?=C2=A0

Again, the builds seem= to be ok and working, but I don=E2=80=99t want to hit some =E2=80=98gotcha= =E2=80=99 if I could avoid it.=C2=A0

<= /div>

On Apr 13, 2016, at 7:15 = AM, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:

Hi,

I am not sure this helps.

we use Spark 1.6 and Hive 2. I also use JDBC = (beeline for Hive)=C2=A0 plus Oracle and Sybase. They all work fine.
<= div>

HTH


On 12 April 2016 at 23:42, Michael Segel <msegel_hadoop@hotmail.com> wrote:
=
Hi,=C2=A0
This is probably a silly question on my part=E2=80=A6=C2= =A0

I=E2=80=99m looking at the latest (spark 1.6.1= release) and would like to do a build w Hive and JDBC support.=C2=A0
=

From the documentation, I see two things that make me s= cratch my head.

1) Scala 2.11=C2=A0
&quo= t;Spark does not yet support its JDBC component for Scala 2.11.=E2=80=9D

So if we want to use JDBC, don=E2=80=99t use Scala 2= .11.x (in this case its 2.11.8)

2) Hive Support
"To enable Hive integration for Spark SQL along with its JDBC = server and CLI, add the -Phive and Phive-thriftserver profiles to= your existing build options. By default Spark will build with Hive 0.13.1 bindings.=E2=80=9D
<= br>
So if we=E2=80=99re looking at a later release of Hive=E2=80= =A6 lets say 1.1.x =E2=80=A6 still use the -Phive and Phive-thriftserver . = Is there anything else we should consider?=C2=A0

J= ust asking because I=E2=80=99ve noticed that this part of the documentation= hasn=E2=80=99t changed much over the past releases.=C2=A0

Thanks in Advance,=C2=A0

-Mike
=


--94eb2c048cb4453c67053061545d--