Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EAE6318E96 for ; Sat, 28 Nov 2015 20:53:08 +0000 (UTC) Received: (qmail 89763 invoked by uid 500); 28 Nov 2015 20:53:06 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 89699 invoked by uid 500); 28 Nov 2015 20:53:06 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 89689 invoked by uid 99); 28 Nov 2015 20:53:06 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Nov 2015 20:53:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id AA91AC05AC for ; Sat, 28 Nov 2015 20:53:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.002 X-Spam-Level: ***** X-Spam-Status: No, score=5.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera-com.20150623.gappssmtp.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id capdtS2MzTSj for ; Sat, 28 Nov 2015 20:52:51 +0000 (UTC) Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id EB60E2026C for ; Sat, 28 Nov 2015 20:52:50 +0000 (UTC) Received: by wmvv187 with SMTP id v187so108395601wmv.1 for ; Sat, 28 Nov 2015 12:52:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=5uulS2f9qDeB8tmcRhU2IBHq/n6tOLWvbyARRH1IT7A=; b=cc0o/ViCDh8JAUeoQF+Wlg88E0+s3QdZf8wwKggP+mJtcaN7Pxg71bDBFWrbBjZlJa 8KRi3mj1l5Md++kr+ruZHsCe/Oy1QsIqpXvscRmTKEpRV2tCAjNp9sQIJW/PciRbCFJ3 1mn1V31edNs25GSlCe9j8FsQ4RWIfvxThUcDXlfmOZlro6TEHoeLW1+oiiVzYCg792cj UutkMar4l7Ki3b7ETCBGucosf1qvTiS2W2kFyJQjxS5wLTpPDg5k2qMmgitWaHJEhfCA wPE1oCScIummuMcq+D0gK2mx9aUXUj5CcrcWBOFGp79L4XmnvdIqkj8EJQHDdLKm1LqF JppA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=5uulS2f9qDeB8tmcRhU2IBHq/n6tOLWvbyARRH1IT7A=; b=Wxq+1tEJByjth55Ps64pNrssElxeOyAMGnXWT82z+Sn9aqZ3TTgGT4dkzULqBqzqLg ZjVr7S948naBBvGGctRcZ7cS76A8u84Iz2U0jp4mkfRfr1aBh6Qpfn03ix7uBfuZXu0y 0n54cVyGAWYd24Lt5s0LxGM6I9gJJuIl6I4Xwzw5FGMWKYob5VPB0YTGaHEMUqCxRUTg payvrzTMmyn4Rzd4tCfJanX07AlyGnaC0xAyeJSpnb5WZZpcfMS9MnxT6n/ZfB6Al17K DV9FTGWY6R1Yseb4jSj4/5GvvGq1/91CbnwYeauRuPIMh41Om/Zvie0QIls41HPEjTj2 6CZA== X-Gm-Message-State: ALoCoQmWXRYI9AASAOkrUlkUfQmfH/fUt3Uc/7UDDFYQ9bKe9aQmRE5tylSjnKmRlpheQSVKlYK9 MIME-Version: 1.0 X-Received: by 10.194.209.235 with SMTP id mp11mr73739048wjc.53.1448743969588; Sat, 28 Nov 2015 12:52:49 -0800 (PST) Received: by 10.28.177.85 with HTTP; Sat, 28 Nov 2015 12:52:49 -0800 (PST) In-Reply-To: <000c01d12a02$ea2a58c0$be7f0a40$@peridale.co.uk> References: <0e7601d12974$799e07c0$6cda1740$@peridale.co.uk> <000c01d12a02$ea2a58c0$be7f0a40$@peridale.co.uk> Date: Sat, 28 Nov 2015 12:52:49 -0800 Message-ID: Subject: Re: Answers to recent questions on Hive on Spark From: Xuefu Zhang To: "user@hive.apache.org" Cc: "dev@hive.apache.org" Content-Type: multipart/alternative; boundary=047d7b33db76adb49b05259ffdcd --047d7b33db76adb49b05259ffdcd Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable You should be able to set that property as any other Hive property: just do "set hive.spark.client.server.address=3Dxxx;" before you start a query. Mak= e sure that you can reach this server address from your nodemanager nodes because they are where the remote driver runs. The driver needs to connect back to HS2. Sometimes firewall may blocks the access, causing the error you seen. Thanks, Xuefu On Sat, Nov 28, 2015 at 9:33 AM, Mich Talebzadeh wrote: > Hi Xuefu, > > > > Thanks for the response. I did the changes as requested (coping the > assembly jar file from build to $HIVE_HOME/lib). I will give full respons= e > when I get the debug outpout > > > > In summary when I ran the sql query from Hive and expected Spark to act a= s > execution engine, it came back with client connection error. > > > > Cruically I noticed that it was trying to connect to eth1 (the internet > connection) as opposed to eth0 (the local network. This host has two > Ethernet cards one for local area network and the other for linternet > (directly no proxy) > > > > It suggested that I can change the address using the configuration > parameter hive.spark.client.server.address > > > > Now I don=E2=80=99t seem to be able to set it up in hive-site.xml or as a= set > parameter in hive prompt itself! > > > > Any hint would be appreciated or any work around? > > > > Regards, > > > > Mich > > > > *From:* Xuefu Zhang [mailto:xzhang@cloudera.com] > *Sent:* 28 November 2015 04:35 > *To:* user@hive.apache.org > *Cc:* dev@hive.apache.org > *Subject:* Re: Answers to recent questions on Hive on Spark > > > > Okay. I think I know what problem you have now. To run Hive on Spark, > spark-assembly.jar is needed and it's also recommended that you have a > spark installation (identified by spark.home) on the same host where HS2 = is > running. You only need spark-assembly.jar in HS2's /lib directory. Other > than those, Hive on Spark doesn't have any other dependency at service > level. On the job level, Hive on Spark jobs of course run on a spark > cluster, which could be standalone, yarn-cluster, etc. However, how you g= et > the binaries for your spark cluster and how you start them is completely > independent of Hive. > > Thus, you only need to build the spark-assembly.jar w/o HIve and put it i= n > Hive's /lib directory. The one in the existing spark build may contain Hi= ve > classes and that's why you need to build your own. Your spark installatio= n > can still have a jar that's different from what you build for Hive on > Spark. Your spark.home can still point to your existing spark installatio= n. > In fact, Hive on Spark only needs spark-submit from your Spark > installation. Therefore, you should be okay even if your spark installati= on > contains Hive classes. > > By following this, I'm sure you will get your Hive on Spark to work. > Depending on the Hive version that your spark installation contains, you > may have problem with spark applications such as SparkSQL, but it shouldn= 't > be a concern if you decide that you use Hive in Hive. > > Let me know if you are still confused. > > Thanks, > > Xuefu > > > > On Fri, Nov 27, 2015 at 4:34 PM, Mich Talebzadeh > wrote: > > Hi, > > > > Thanks for heads up and comments. > > > > Sounds like when it comes to using spark as the execution engine for Hive= , > we are in no man=E2=80=99s land so to speak. I have opened questions in b= oth Hive > and Spark user forums. Not much of luck for reasons that you alluded to. > > > > Ok just to clarify the prebuild version of spark (as opposed get the > source code and build your spec) works fine for me. > > > > Components are > > > > hadoop version > > Hadoop 2.6.0 > > > > hive --version > > Hive 1.2.1 > > > > Spark > > version 1.5.2 > > > > It does what it says on the tin. For example I can start the master node > OK start-master.sh. > > > > > > Spark Command: */usr/java/latest/bin/java -cp > /usr/lib/spark_1.5.2_bin/sbin/../conf/:/usr/lib/spark_1.5.2_bin/lib/spark= -assembly-1.5.2-hadoop2.6.0.jar:/usr/lib/spark_1.5.2_bin/lib/datanucleus-co= re-3.2.10.jar:/usr/lib/spark_1.5.2_bin/lib/datanucleus-api-jdo-3.2.6.jar:/u= sr/lib/spark_1.5.2_bin/lib/datanucleus-rdbms-3.2.9.jar:/home/hduser/hadoop-= 2.6.0/etc/hadoop/ > -Xms1g -Xmx1g -XX:MaxPermSize=3D256m org.apache.spark.deploy.master.Maste= r > --ip 127.0.0.1 --port 7077 --webui-port 8080* > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 15/11/28 00:05:23 INFO master.Master: Registered signal handlers for > [TERM, HUP, INT] > > 15/11/28 00:05:23 WARN util.Utils: Your hostname, rhes564 resolves to a > loopback address: 127.0.0.1; using 50.140.197.217 instead (on interface > eth0) > > 15/11/28 00:05:23 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind > to another address > > 15/11/28 00:05:24 WARN util.NativeCodeLoader: Unable to load native-hadoo= p > library for your platform... using builtin-java classes where applicable > > 15/11/28 00:05:24 INFO spark.SecurityManager: Changing view acls to: hdus= er > > 15/11/28 00:05:24 INFO spark.SecurityManager: Changing modify acls to: > hduser > > 15/11/28 00:05:24 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(hduser); users with modify permissions: Set(hduser) > > 15/11/28 00:05:25 INFO slf4j.Slf4jLogger: Slf4jLogger started > > 15/11/28 00:05:25 INFO Remoting: Starting remoting > > 15/11/28 00:05:25 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkMaster@127.0.0.1:7077] > > 15/11/28 00:05:25 INFO util.Utils: Successfully started service > 'sparkMaster' on port 7077. > > 15/11/28 00:05:25 INFO master.Master: Starting Spark master at spark:// > 127.0.0.1:7077 > > 15/11/28 00:05:25 INFO master.Master: Running Spark version 1.5.2 > > 15/11/28 00:05:25 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/11/28 00:05:25 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:8080 > > 15/11/28 00:05:25 INFO util.Utils: Successfully started service 'MasterUI= ' > on port 8080. > > 15/11/28 00:05:25 INFO ui.MasterWebUI: Started MasterWebUI at > http://50.140.197.217:8080 > > 15/11/28 00:05:25 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/11/28 00:05:25 INFO server.AbstractConnector: Started > SelectChannelConnector@rhes564:6066 > > 15/11/28 00:05:25 INFO util.Utils: Successfully started service on port > 6066. > > 15/11/28 00:05:25 INFO rest.StandaloneRestServer: Started REST server for > submitting applications on port 6066 > > 15/11/28 00:05:25 INFO master.Master: I have been elected leader! New > state: ALIVE > > > > However, I cannot use spark in place of MapReduce engine with this build. > It fails > > > > The instruction says download the source code for spark and build it by > excluding Hive jar files so that you can use spark as the execution engin= e > > > > Ok > > > > I downloaded spark 1.5.2 source code and used the following to create the > tarred and zipped file > > > > ./make-distribution.sh --name "hadoop2-without-hive" --tgz > "-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided" > > > > After unpacking the file, I attempted to start the master node as above s= tart-master.sh, > However, regrettably it fails with the following error > > > > > > *Spark Command: /usr/java/latest/bin/java -cp > /usr/lib/spark_1.5.2_build/sbin/../conf/:/usr/lib/spark_1.5.2_build/lib/* > *spark-assembly-1.5.2-hadoop2.4.0.jar**:/home/hduser/hadoop-2.6.0/etc/had= oop/ > -Xms1g -Xmx1g -XX:MaxPermSize=3D256m org.apache.spark.deploy.master.Maste= r > --ip 127.0.0.1 --port 7077 --webui-port 8080* > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logg= er > > at java.lang.Class.getDeclaredMethods0(Native Method) > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2521) > > at java.lang.Class.getMethod0(Class.java:2764) > > at java.lang.Class.getMethod(Class.java:1653) > > at > sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494) > > at > sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486) > > Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > ... 6 more > > > > > > I believe the problem lies in *spark-assembly-1.5.2-hadoop2.4.0.jar *file= . > Case in point, if I copy the jar file *spark-assembly-1.5.2-hadoop2.6.0.j= ar > *to the lib directory above , I can start the master node. > > > > hduser@rhes564::/usr/lib/spark_1.5.2_build/lib> mv > spark-assembly-1.5.2-hadoop2.4.0.jar > spark-assembly-1.5.2-hadoop2.4.0.jar_old > > hduser@rhes564::/usr/lib/spark_1.5.2_build/lib> cp > /usr/lib/spark_1.5.2_bin/lib/spark-assembly-1.5.2-hadoop2.6.0.jar . > > > > hduser@rhes564::/usr/lib/spark_1.5.2_build/lib> cd ../sbin > > hduser@rhes564::/usr/lib/spark_1.5.2_build/sbin> start-master.sh > > starting org.apache.spark.deploy.master.Master, logging to > /usr/lib/spark_1.5.2_build/sbin/../logs/spark-hduser-org.apache.spark.dep= loy.master.Master-1-rhes564.out > > hduser@rhes564::/usr/lib/spark_1.5.2_build/sbin> cat > /usr/lib/spark_1.5.2_build/sbin/../logs/spark-hduser-org.apache.spark.dep= loy.master.Master-1-rhes564.out > > *Spark Command: /usr/java/latest/bin/java -cp > /usr/lib/spark_1.5.2_build/sbin/../conf/:/usr/lib/spark_1.5.2_build/lib/* > *spark-assembly-1.5.2-hadoop2.6.0.jar**:/home/hduser/hadoop-2.6.0/etc/had= oop/ > -Xms1g -Xmx1g -XX:MaxPermSize=3D256m org.apache.spark.deploy.master.Maste= r > --ip 50.140.197.217 --port 7077 --webui-port 8080* > > *=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D* > > 15/11/28 00:31:24 INFO master.Master: Registered signal handlers for > [TERM, HUP, INT] > > 15/11/28 00:31:25 WARN util.NativeCodeLoader: Unable to load native-hadoo= p > library for your platform... using builtin-java classes where applicable > > 15/11/28 00:31:25 INFO spark.SecurityManager: Changing view acls to: hdus= er > > 15/11/28 00:31:25 INFO spark.SecurityManager: Changing modify acls to: > hduser > > 15/11/28 00:31:25 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(hduser); users with modify permissions: Set(hduser) > > 15/11/28 00:31:25 INFO slf4j.Slf4jLogger: Slf4jLogger started > > 15/11/28 00:31:26 INFO Remoting: Starting remoting > > 15/11/28 00:31:26 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkMaster@50.140.197.217:7077] > > 15/11/28 00:31:26 INFO util.Utils: Successfully started service > 'sparkMaster' on port 7077. > > 15/11/28 00:31:26 INFO master.Master: Starting Spark master at spark:// > 50.140.197.217:7077 > > 15/11/28 00:31:26 INFO master.Master: Running Spark version 1.5.2 > > 15/11/28 00:31:26 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/11/28 00:31:26 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:8080 > > 15/11/28 00:31:26 INFO util.Utils: Successfully started service 'MasterUI= ' > on port 8080. > > 15/11/28 00:31:26 INFO ui.MasterWebUI: Started MasterWebUI at > http://50.140.197.217:8080 > > 15/11/28 00:31:26 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/11/28 00:31:26 INFO server.AbstractConnector: Started > SelectChannelConnector@c-50-140-197-217.hsd1.fl.comcast.net:6066 > > 15/11/28 00:31:26 INFO util.Utils: Successfully started service on port > 6066. > > 15/11/28 00:31:26 INFO rest.StandaloneRestServer: Started REST server for > submitting applications on port 6066 > > 15/11/28 00:31:27 INFO master.Master: I have been elected leader! New > state: ALIVE > > > > Thanks again. > > > > > > Mich Talebzadeh > > > > *Sybase ASE 15 Gold Medal Award 2008* > > A Winning Strategy: Running the most Critical Financial Data on ASE 15 > > > http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-0919= 08.pdf > > Author of the books* "A Practitioner=E2=80=99s Guide to Upgrading to Syba= se ASE > 15", ISBN 978-0-9563693-0-7*. > > co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN > 978-0-9759693-0-4* > > *Publications due shortly:* > > *Complex Event Processing in Heterogeneous Environments*, ISBN: > 978-0-9563693-3-8 > > *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, vol= ume > one out shortly > > > > http://talebzadehmich.wordpress.com > > > > NOTE: The information in this email is proprietary and confidential. This > message is for the designated recipient only, if you are not the intended > recipient, you should destroy it immediately. Any information in this > message shall not be understood as given or endorsed by Peridale Technolo= gy > Ltd, its subsidiaries or their employees, unless expressly so stated. It = is > the responsibility of the recipient to ensure that this email is virus > free, therefore neither Peridale Ltd, its subsidiaries nor their employee= s > accept any responsibility. > > > > *From:* Xuefu Zhang [mailto:xzhang@cloudera.com] > *Sent:* 27 November 2015 18:12 > *To:* user@hive.apache.org; dev@hive.apache.org > *Subject:* Answers to recent questions on Hive on Spark > > > > Hi there, > > There seemed an increasing interest in Hive On Spark From the Hive users. > I understand that there have been a few questions or problems reported an= d > I can see some frustration sometimes. It's impossible for Hive on Spark > team to respond every inquiry even thought we wish we could. However, the= re > are a few items to be noted: > > 1. Hive on Spark is being tested as part of Precommit test. > > 2. Hive on Spark is supported in some distributions such as CDH. > > 3. I tried a couple of days ago with latest master and branch-1, and they > all worked with my Spark 1.5 build. > > Therefore, if you are facing some problem, it's likely due to your setup. > Please refer to Wiki on how to do it right. Nevertheless, I have a few > suggestions here: > > 1. Start with simple. Try out a CDH sandbox or distribution first and to > see it works in action before building your own. Comparing with your setu= p > may give you some clues. > > 2. Try with spark.master=3Dlocal first, making sure that you have all the > necessary dependent jars, and then move to your production setup. Please > note that yarn-cluster is recommended and mesos is not supported. I tried > both yarn-cluster and local-cluster and both worked for me. > > 3. Check logs beyond hive.log such as spark log, and yarn-log to get more > error messages. > > When you report your problem, please provide as much info as possible, > such as your platform, your builds, your configurations, and relevant log= s > so that others can reproduce. > > Please note that we are not in a good position to answer questions with > respect to Spark itself, such as spark-shell. Not only is that beyond the > scope of Hive on Scope, but also the team may not have the expertise to > give your meaningful answers. One thing to emphasize. When you build your > spark jar, don't include Hive, as it's very likely there is a version > mismatch. Again, a distribution may have solve the problem for you if you > like to give it a try. > > Hope this helps. > > Thanks, > > Xuefu > > > --047d7b33db76adb49b05259ffdcd Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
You should be a= ble to set that property as any other Hive property: just do "set hive= .spark.client.server.address=3Dxxx;" before you start a query. Make su= re that you can reach this server address from your nodemanager nodes becau= se they are where the remote driver runs. The driver needs to connect back = to HS2. Sometimes firewall may blocks the access, causing the error you see= n.

Thank= s,
Xuefu<= /span>

On Sat, Nov 28, 2015 at 9:33 AM, Mich Talebzadeh <mi= ch@peridale.co.uk> wrote:
<= div link=3D"blue" vlink=3D"purple" lang=3D"EN-GB">

Hi Xuefu,

= =C2=A0

Thanks for the response. I did the changes as requested (coping th= e assembly jar file from build to $HIVE_HOME/lib). I will give full respons= e when I get the debug outpout

=C2=A0

In summary when I ran the sql query from Hive and expected Spar= k to act as execution engine, it came back with client connection error.=

=C2=A0

Cruically I noticed t= hat it was trying to connect to eth1 (the internet connection) as opposed t= o eth0 (the local network. This host has two Ethernet cards one for local a= rea network and the other for linternet (directly no proxy)

=C2=A0<= /p>

It suggested that I can change the= address using the configuration parameter hive.spark.client.se= rver.address

=C2= =A0

Now I don=E2=80=99t seem to be= able to set it up in hive-site.xml or as a set parameter in hive prompt it= self!

=C2=A0<= /span>

Any hint would be appreciated or any work= around?

=C2=A0

Regards,

=C2=A0

M= ich

=C2=A0=

From: Xuefu Zhang [mailto:xzhang@cloudera.com]
Sent: 28 November 2015 04:35To: user@= hive.apache.org
Cc: dev@hive.apache.org
Subject: Re: Answers to r= ecent questions on Hive on Spark

=C2=A0

Okay. I think = I know what problem you have now. To run Hive on Spark, spark-assembly.jar = is needed and it's also recommended that you have a spark installation = (identified by spark.home) on the same host where HS2 is running. You only = need spark-assembly.jar in HS2's /lib directory. Other than those, Hive= on Spark doesn't have any other dependency at service level. On the jo= b level, Hive on Spark jobs of course run on a spark cluster, which could b= e standalone, yarn-cluster, etc. However, how you get the binaries for your= spark cluster and how you start them is completely independent of Hive.=

= Thus, you only need to build the spark-assembly.jar w/o HIve and put it in = Hive's /lib directory. The one in the existing spark build may contain = Hive classes and that's why you need to build your own. Your spark inst= allation can still have a jar that's different from what you build for = Hive on Spark. Your spark.home can still point to your existing spark insta= llation. In fact, Hive on Spark only needs spark-submit from your Spark ins= tallation. Therefore, you should be okay even if your spark installation co= ntains Hive classes.

By following this, I'm sure you will get your Hi= ve on Spark to work. Depending on the Hive version that your spark installa= tion contains, you may have problem with spark applications such as SparkSQ= L, but it shouldn't be a concern if you decide that you use Hive in Hiv= e.

Let me know if you are still confused.

Thanks,

Xuefu<= u>

=C2=A0

=

On Fri, Nov 27, 2015 at 4:34 PM, Mich Talebzadeh <= ;mich@peridale.co.= uk> wrote:

Hi,

=C2=A0

Tha= nks for heads up and comments.

=C2=A0

Sounds like when it = comes to using spark as the execution engine for Hive, we are in no man=E2= =80=99s land so to speak. I have opened questions in both Hive and Spark us= er forums. Not much of luck for reasons that you alluded to.<= u>

=C2=A0

Ok just to clarify the prebuild version of spark (as opposed get = the source code and build your spec) works fine for me.

=C2=A0

Components are

=C2=A0

hadoop version<= /u>

Hadoop 2.6.0=

=C2=A0

hive --version

Hive 1.2.1

=C2=A0

Spark

version 1.5.2

=C2=A0

It d= oes what it says on the tin. For example I can start the master node OK start-master.sh.

=C2=A0

=C2=A0

Spark Command: /usr= /java/latest/bin/java -cp /usr/lib/spark_1.5.2_bin/sbin/../conf/:/usr/lib/s= park_1.5.2_bin/lib/spark-assembly-1.5.2-hadoop2.6.0.jar:/usr/lib/spark_1.5.= 2_bin/lib/datanucleus-core-3.2.10.jar:/usr/lib/spark_1.5.2_bin/lib/datanucl= eus-api-jdo-3.2.6.jar:/usr/lib/spark_1.5.2_bin/lib/datanucleus-rdbms-3.2.9.= jar:/home/hduser/hadoop-2.6.0/etc/hadoop/ -Xms1g -Xmx1g -XX:MaxPermSize=3D2= 56m org.apache.spark.deploy.master.Master --ip 127.0.0.1 --port 7077 --webu= i-port 8080

=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

15/11/28 00:05:23 INFO master.Master: Registered signal ha= ndlers for [TERM, HUP, INT]

= 15/11/28 00:05:23 WARN util.Utils: Your hostname, rhes564 resolves to = a loopback address: 127.0.0.1; using 50.140.197.217 instead (on interface e= th0)

15/11/28 00:05:23 = WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address<= /span>

15/11/28 00:05:24 WARN = util.NativeCodeLoader: Unable to load native-hadoop library for your platfo= rm... using builtin-java classes where applicable

<= p class=3D"MsoNormal">15/11/28 00:05:24 INFO spark.SecurityManager: Ch= anging view acls to: hduser

= 15/11/28 00:05:24 INFO spark.SecurityManager: Changing modify acls to:= hduser

15/11/28 00:05:= 24 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui= acls disabled; users with view permissions: Set(hduser); users with modify= permissions: Set(hduser)

15/11/28 00:05:25 INFO slf4j.Slf4jLogger: Slf4jLogger started<= /u>

15/11/28 00:05:25 INFO Remoting: = Starting remoting

15/11= /28 00:05:25 INFO Remoting: Remoting started; listening on addresses :[akka= .tcp://spar= kMaster@127.0.0.1:7077]

= 15/11/28 00:05:25 INFO util.Utils: Successfully started service 's= parkMaster' on port 7077.

15/11/28 00:05:25 INFO master.Master: Starting Spark master at spark= ://127.0.0.1:7077

15/11/28 00:05:25 INFO m= aster.Master: Running Spark version 1.5.2

15/11/28 00:05:25 INFO server.Server: jetty-8.y.z-SNAPS= HOT

15/11/28 00:05:25 I= NFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8080=

15/11/28 00:05:25 INFO= util.Utils: Successfully started service 'MasterUI' on port 8080.<= /span>

15/11/28 00:05:25 INFO = ui.MasterWebUI: Started MasterWebUI at http://50.140.197.217:8080

15/11/28 00:05:25 INFO server.Server: jetty-8.= y.z-SNAPSHOT

15/11/28 0= 0:05:25 INFO server.AbstractConnector: Started SelectChannelConnector@rhes5= 64:6066

15/11/28 00:05:= 25 INFO util.Utils: Successfully started service on port 6066.

15/11/28 00:05:25 INFO rest.Standal= oneRestServer: Started REST server for submitting applications on port 6066=

15/11/28 00:05:25 INFO= master.Master: I have been elected leader! New state: ALIVE<= u>

=C2=A0

However, I cannot use spark in place of MapReduce engine with thi= s build. It fails

=C2=A0

The instruction says download th= e source code for spark and build it by excluding Hive jar files so that yo= u can use spark as the execution engine

=C2=A0

Ok

=C2=A0

I downloaded spark 1.5.2 source code and used the following= to create the tarred and zipped file

=C2=A0

./make= -distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn= ,hadoop-provided,hadoop-2.4,parquet-provided"

=

=C2=A0

= After unpacking the file, I attempted to start the master node as above start-master.sh, However, regrettably it fails with the foll= owing error

=C2=A0<= /u>

=C2=A0

Spark Command: /usr/java/latest/bin/java -cp /u= sr/lib/spark_1.5.2_build/sbin/../conf/:/usr/lib/spark_1.5.2_build/lib/spark-assembly-1.5.2-hadoop2.4.0.jar:/home/= hduser/hadoop-2.6.0/etc/hadoop/ -Xms1g -Xmx1g -XX:MaxPermSize=3D256m org.ap= ache.spark.deploy.master.Master --ip 127.0.0.1 --port 7077 --webui-port 808= 0

=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D

= Exception in thread "main" java.lang.NoClassDefFoundError: or= g/slf4j/Logger

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Class.getDeclaredMethods0= (Native Method)

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Class.privateGetDeclaredM= ethods(Class.java:2521)

= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Class.getMethod0(Cl= ass.java:2764)

=C2=A0 = =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at java.lang.Class.getMethod(Class.java= :1653)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.launcher.LauncherHelper.getMainMethod(La= uncherHelper.java:494)

= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.launcher.LauncherHelper.c= heckAndLoadMain(LauncherHelper.java:486)

Caused by: java.lang.ClassNotFoundException: org.slf4j.L= ogger

=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 at java.net.URLClassLoader$1.run(URLClassLoader.ja= va:366)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.net.URLClassLoader$1.run(URLClassLoader= .java:355)

=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.security.AccessController.doPrivileg= ed(Native Method)

=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.net.URLClassLoader.findClass(U= RLClassLoader.java:354)

= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.ClassLoader.loadCla= ss(ClassLoader.java:424)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.misc.Launcher$AppClassLo= ader.loadClass(Launcher.java:308)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.ClassLoad= er.loadClass(ClassLoader.java:357)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ... 6 more =

=C2=A0

<= p class=3D"MsoNormal">=C2=A0

I believe the problem lies in spark-assembly= -1.5.2-hadoop2.4.0.jar file. Case in point, if I copy= the jar file spark-assembly-1.5.2-hadoop2.6.0.jar to the lib directory above , I can start the master node.=

=C2=A0

hduser@rhes564::/usr/lib/spark_1.5.2_build/lib&g= t; mv spark-assembly-1.5.2-hadoop2.4.0.jar spark-assembly-1.5.2-hadoop2.4.0= .jar_old

hduse= r@rhes564::/usr/lib/spark_1.5.2_build/lib> cp /usr/lib/spark_1.5.2_b= in/lib/spark-assembly-1.5.2-hadoop2.6.0.jar .

=C2=A0

hduser@rhes564::/usr/lib/spark_1.5.2_build/lib> cd ../sb= in

hduser@rhe= s564::/usr/lib/spark_1.5.2_build/sbin> start-master.sh=

starting org.apache.spark.deploy.mas= ter.Master, logging to /usr/lib/spark_1.5.2_build/sbin/../logs/spark-hduser= -org.apache.spark.deploy.master.Master-1-rhes564.out

hduser@rhes564::/usr/lib/spark_1.5.= 2_build/sbin> cat /usr/lib/spark_1.5.2_build/sbin/../logs/spark-hdus= er-org.apache.spark.deploy.master.Master-1-rhes564.out=

Spark Command: /usr/java/latest/bin/java= -cp /usr/lib/spark_1.5.2_build/sbin/../conf/:/usr/lib/spark_1.5.2_build/li= b/spark-assembly-1.5.2-hadoop2.6.0.jar:/home/hduser/hadoop-2.6.0/etc/hadoop/ -Xms1g -Xmx1g -XX:MaxPermSize=3D256= m org.apache.spark.deploy.master.Master --ip 50.140.197.217 --port 7077 --w= ebui-port 8080

= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

<= p class=3D"MsoNormal">15/11/28 00:31:24 INFO master.Master: Registered= signal handlers for [TERM, HUP, INT]

15/11/28 00:31:25 WARN util.NativeCodeLoader: Unable to load= native-hadoop library for your platform... using builtin-java classes wher= e applicable

15/11/28 0= 0:31:25 INFO spark.SecurityManager: Changing view acls to: hduser=

15/11/28 00:31:25 INFO spark.Sec= urityManager: Changing modify acls to: hduser

15/11/28 00:31:25 INFO spark.SecurityManager: Securi= tyManager: authentication disabled; ui acls disabled; users with view permi= ssions: Set(hduser); users with modify permissions: Set(hduser)

15/11/28 00:31:25 INFO slf4j.Slf4j= Logger: Slf4jLogger started

= 15/11/28 00:31:26 INFO Remoting: Starting remoting

15/11/28 00:31:26 INFO Remoting: Remoting = started; listening on addresses :[akka.tcp://sparkMaster@50.140.197.217:7077]=

15/11/28 00:31:26 INFO= util.Utils: Successfully started service 'sparkMaster' on port 707= 7.

15/11/28 00:31:26 IN= FO master.Master: Starting Spark master at spark://50.140.197.217:7077

15/11/28 00:31:26 INFO master.Master: Run= ning Spark version 1.5.2

15/11/28 00:31:26 INFO server.Abstrac= tConnector: Started SelectChannelConnector@0.0.0.0:8080<= /u>

15/11/28 00:31:26 INFO util.Utils: Succe= ssfully started service 'MasterUI' on port 8080.

15/11/28 00:31:26 INFO ui.MasterWebUI: St= arted MasterWebUI at http://50.140.197.217:8080

15/11/28 00:31:26 INFO server.Server: jetty-8.y.z-SNAPSHOT

15/11/28 00:31:26 INFO serve= r.AbstractConnector: Started SelectChannelConnec= tor@c-50-140-197-217.hsd1.fl.comcast.net:6066

<= p class=3D"MsoNormal">15/11/28 00:31:26 INFO util.Utils: Successfully = started service on port 6066.

15/11/28 00:31:26 INFO rest.StandaloneRestServer: Started REST serve= r for submitting applications on port 6066

15/11/28 00:31:27 INFO master.Master: I have been elect= ed leader! New state: ALIVE

= =C2=A0

Thanks again.

=C2=A0

<= p class=3D"MsoNormal">=C2=A0

= M= ich Talebzadeh

=C2=A0

Sybase ASE 15 Gold Medal Award = 2008

A Winning Strategy: Running the most Critical Finan= cial Data on ASE 15

http://login.sybase.com/files/Product_Overviews/AS= E-Winning-Strategy-091908.pdf

Author of the books "= A Practitioner=E2=80=99s Guide to Upgrading to Sybase ASE 15", ISBN 97= 8-0-9563693-0-7.

co-author "Sybase Transact SQL G= uidelines Best Practices", ISBN 978-0-9759693-0-4=

Publications due shortly:

Complex Event Pro= cessing in Heterogeneous Environments, ISBN: 978-0-95= 63693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978= -0-9563693-1-4, volume one out shortly

= =C2=A0

http://talebzadehmich.wordpress.com

=C2=A0

NOTE: The information in this email is proprietary and confident= ial. This message is for the designated recipient only, if you are not the = intended recipient, you should destroy it immediately. Any information in t= his message shall not be understood as given or endorsed by Peridale Techno= logy Ltd, its subsidiaries or their employees, unless expressly so stated. = It is the responsibility of the recipient to ensure that this email is viru= s free, therefore neither Peridale Ltd, its subsidiaries nor their employee= s accept any responsibility.

= =C2=A0

Fr= om: Xuefu Zhang [mailto:xzhang@cloudera.com]
Sent: = 27 November 2015 18:12
To: user@hive.apache.org; dev@hive.apache.org
Subject: Answ= ers to recent questions on Hive on Spark

=

=C2=A0

Hi there,

Th= ere seemed an increasing interest in Hive On Spark From the Hive users. I u= nderstand that there have been a few questions or problems reported and I c= an see some frustration sometimes. It's impossible for Hive on Spark te= am to respond every inquiry even thought we wish we could. However, there a= re a few items to be noted:

1= . Hive on Spark is being tested as part of Precommit test.

2. Hive on Spark is supported in some distrib= utions such as CDH.

3. I tried a couple of days ago with latest master an= d branch-1, and they all worked with my Spark 1.5 build.

<= /div>

Therefore, if yo= u are facing some problem, it's likely due to your setup. Please refer = to Wiki on how to do it right. Nevertheless, I have a few suggestions here:=

1. Start with simple. Try ou= t a CDH sandbox or distribution first and to see it works in action before = building your own. Comparing with your setup may give you some clues.

2. Try with spark.master=3Dloc= al first, making sure that you have all the necessary dependent jars, and t= hen move to your production setup. Please note that yarn-cluster is recomme= nded and mesos is not supported. I tried both yarn-cluster and local-cluste= r and both worked for me.

3. Check logs beyond hive.log such as spar= k log, and yarn-log to get more error messages.

When you report your= problem, please provide as much info as possible, such as your platform, y= our builds, your configurations, and relevant logs so that others can repro= duce.

Please note that we are not in a good position to answer quest= ions with respect to Spark itself, such as spark-shell. Not only is that be= yond the scope of Hive on Scope, but also the team may not have the experti= se to give your meaningful answers. One thing to emphasize. When you build = your spark jar, don't include Hive, as it's very likely there is a = version mismatch. Again, a distribution may have solve the problem for you = if you like to give it a try.

Hope this helps.

Thanks,

Xuefu

<= /div>

=C2=A0


--047d7b33db76adb49b05259ffdcd--