hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xuefu Zhang <xzh...@cloudera.com>
Subject Re: Getting error when trying to start master node after building spark 1.3
Date Sun, 06 Dec 2015 14:14:48 GMT
That basically says that snappy isn't working properly on your box. You can
forget about that for now by running:

set spark.io.compression.codec=org.apache.spark.io.LZFCompressionCodec;


On Sat, Dec 5, 2015 at 1:45 PM, Mich Talebzadeh <mich@peridale.co.uk> wrote:

> Great stuff.
>
>
>
> Built the source code for 1.3.1 and generated
> spark-assembly-1.3.1-hadoop2.4.0.jar
>
>
>
> jar tvf spark-assembly-1.3.1-hadoop2.4.0.jar|grep hive | grep -i -v Archive
>
>
>
> so no hive there
>
>
>
> Downloaded prebuilt spark 1.3.1 and started master and slave OK
>
>
>
> Started hive as usual in debug mode  and did a simple select count(1) from
> t;
>
>
>
> Spark app started OK
>
>
>
> hduser@rhes564::/usr/lib/spark-1.3.1-bin-hadoop2.6/logs>
>
>
>
> -rw-r--r-- 1 hduser hadoop  31562 Dec  5 21:18
> spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out
>
> -rw-r--r-- 1 hduser hadoop  19684 Dec  5 21:18
> spark-hduser-org.apache.spark.deploy.worker.Worker-1-rhes564.out
>
> -rwxrwx--- 1 hduser hadoop  60491 Dec  5 21:18
> app-20151205211814-0005.inprogress
>
>
>
> Now I get some library error
>
>
>
> 5/12/05 21:18:16 [stderr-redir-1]: INFO client.SparkClientImpl: Caused by:
> java.lang.UnsatisfiedLinkError: /tmp/snappy-1.0.5-libsnappyjava.so:
> /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by
> /tmp/snappy-1.0.5-libsnappyjava.so)
>
>
>
>
>
> strings /usr/lib/libstdc++.so.6 | grep GLIBC
>
> GLIBCXX_3.4
>
> GLIBCXX_3.4.1
>
> GLIBCXX_3.4.2
>
> GLIBCXX_3.4.3
>
> GLIBCXX_3.4.4
>
> GLIBCXX_3.4.5
>
> GLIBCXX_3.4.6
>
> GLIBCXX_3.4.7
>
> GLIBCXX_3.4.8
>
> GLIBC_2.3
>
> GLIBC_2.0
>
> GLIBC_2.3.2
>
> GLIBC_2.4
>
> GLIBC_2.1
>
> GLIBC_2.1.3
>
> GLIBC_2.2
>
> GLIBCXX_FORCE_NEW
>
>
>
> Looking into sorting this out.
>
>
>
> Mich Talebzadeh
>
>
>
> *Sybase ASE 15 Gold Medal Award 2008*
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
>
>
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE
> 15", ISBN 978-0-9563693-0-7*.
>
> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4*
>
> *Publications due shortly:*
>
> *Complex Event Processing in Heterogeneous Environments*, ISBN:
> 978-0-9563693-3-8
>
> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume
> one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Ltd, its subsidiaries nor their employees
> accept any responsibility.
>
>
>
> *From:* Xuefu Zhang [mailto:xzhang@cloudera.com]
> *Sent:* 04 December 2015 17:47
> *To:* user@hive.apache.org
> *Subject:* Re: FW: Getting error when trying to start master node after
> building spark 1.3
>
>
>
> 1.3.1 is what officially supported by Hive 1.2.1. 1.3.0 might be okay too.
>
>
>
> On Fri, Dec 4, 2015 at 9:34 AM, Mich Talebzadeh <mich@peridale.co.uk>
> wrote:
>
> Appreciated the response. Just to clarify the build will be spark 1.3 and
> the pre-build download will be 1.3. this is the version I am attempting to
> make it work.
>
>
>
> Thanks
>
>
>
> Mich
>
>
>
> *From:* Xuefu Zhang [mailto:xzhang@cloudera.com]
> *Sent:* 04 December 2015 17:03
> *To:* user@hive.apache.org
> *Subject:* Re: FW: Getting error when trying to start master node after
> building spark 1.3
>
>
>
> My last attempt:
>
> 1. Make sure the spark-assembly.jar from your own build doesn't contain
> hive classes, using "jar -tf spark-assembly.jar | grep hive" command. Copy
> it to Hive's /lib directory. After this, you can forget everything about
> this build.
>
> 2. Download prebuilt tarball from Spark download site and deploy it.
> Forget about Hive for a moment. Make sure the cluster comes up and
> functions.
>
> 3. Unset environment variable SPARK_HOME before you start Hive. Start
> Hive, and run "set spark.home=/path/to/spark/dir" command. Then run other
> commands as you did previously when trying hive on spark.
>
>
>
>
>
>
>

Mime
View raw message