hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mich Talebzadeh" <>
Subject The Hive shell and Spark issue
Date Mon, 07 Dec 2015 00:47:08 GMT


Sounds like the issue with Hive and Spark as Hive engine comes from the
following lines in $HIVE_HOME/bin/hive which is


# add Spark assembly jar to the classpath

if [[ -n "$SPARK_HOME" ]]


  sparkAssemblyPath=`ls ${SPARK_HOME}/lib/spark-assembly-*.jar`




As we know Hive will not be able to use Spark with spark-assembly-*.jar
which is located in pre-built spark download. It will not work! For now as a
work-around you need to build Spark from source code and exclude Hive
libraries. Then copy spark-assembly-*.jar file from
want to test Hive with Spark engine.


So either you have to unset the ENV variable $SPARK_HOME when connecting to
Hive CLI or comment out CLASSPATH="${CLASSPATH}:${sparkAssemblyPath}" in


But leaving the file  "spark-assembly-*.jar" in $HIVE_HOME/bin/hive seems to
cause Hive server not to start properly so client connections like beeline
don't seem to work as well.


I am still investigating. 






Mich Talebzadeh


Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15


Author of the books "A Practitioner's Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN:

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
one out shortly




NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.


View raw message