hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: Any clue on this error, Exception in thread "main" java.lang.NoSuchFieldError: SPARK_RPC_CLIENT_CONNECT_TIMEOUT
Date Thu, 03 Dec 2015 23:44:29 GMT
I spoke to Xuefu (Hive dev) and mentioned that this isn't really how
it should be done.

In the meantime, if you can, you should use a Spark package that does
not include Hive classes. There used to be an explicit one for that,
but I can't find it. In the meantime, the tarball that says "pre-built
with user-provided Hadoop" should work for your case.

On Thu, Dec 3, 2015 at 3:41 PM, Mich Talebzadeh <mich@peridale.co.uk> wrote:
> Just noticed that hive shell in 1.2.1 makes a reference to SPARK_HOME if it
> finds it
>
>
>
>
>
> # add Spark assembly jar to the classpath
>
> if [[ -n "$SPARK_HOME" ]]
>
> then
>
>   sparkAssemblyPath=`ls ${SPARK_HOME}/lib/spark-assembly-*.jar`
>
>   CLASSPATH="${CLASSPATH}:${sparkAssemblyPath}"
>
> fi
>
>
>
>
>
> Is this expected?
>
>
>
> Mich Talebzadeh
>
>
>
> Sybase ASE 15 Gold Medal Award 2008
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
>
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15",
> ISBN 978-0-9563693-0-7.
>
> co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4
>
> Publications due shortly:
>
> Complex Event Processing in Heterogeneous Environments, ISBN:
> 978-0-9563693-3-8
>
> Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
> one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus free,
> therefore neither Peridale Ltd, its subsidiaries nor their employees accept
> any responsibility.
>
>
>
> From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
> Sent: 03 December 2015 19:46
> To: user@hive.apache.org; 'Marcelo Vanzin' <vanzin@cloudera.com>
>
>
> Subject: RE: Any clue on this error, Exception in thread "main"
> java.lang.NoSuchFieldError: SPARK_RPC_CLIENT_CONNECT_TIMEOUT
>
>
>
> Hi,
>
>
>
> This is my CLASSPATH which I have simplified running with Hive 1.2.1 and
> generic build Spark 1.3
>
>
>
> unset CLASSPATH
>
> CLASSPATH=$HADOOP_HOME/share/hadoop/common/hadoop-common-2.6.0-tests.jar:$HADOOP_HOME/share/hadoop/common/hadoop-common-2.6.0.jar:hadoop-nfs-2.6.0.jar:$HIVE_HOME/lib:${SPARK_HOME}/lib
>
>
>
> echo $CLASSPATH
>
> export CLASSPATH
>
>
>
>
>
> CLASPPATH IS now
>
>
>
> /home/hduser/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0-tests.jar:/home/hduser/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0.jar:hadoop-nfs-2.6.0.jar:/usr/lib/hive/lib:/usr/lib/spark/lib
>
>
>
> However, I get the error. Does anyone has a working CLASSPATH for this?
>
>
>
>
>
>
>
> .spark.client.RemoteDriver /usr/lib/hive/lib/hive-exec-1.2.1.jar
> --remote-host rhes564 --remote-port 51642 --conf
> hive.spark.client.connect.timeout=1000 --conf
> hive.spark.client.server.connect.timeout=90000 --conf
> hive.spark.client.channel.log.level=null --conf
> hive.spark.client.rpc.max.size=52428800 --conf
> hive.spark.client.rpc.threads=8 --conf hive.spark.client.secret.bits=256
>
> 15/12/03 19:42:51 [stderr-redir-1]: INFO client.SparkClientImpl: Spark
> assembly has been built with Hive, including Datanucleus jars on classpath
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property: hive.spark.client.connect.timeout=1000
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property: hive.spark.client.rpc.threads=8
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property: hive.spark.client.rpc.max.size=52428800
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property: hive.spark.client.secret.bits=256
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl: Warning:
> Ignoring non-spark config property:
> hive.spark.client.server.connect.timeout=90000
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl: 15/12/03
> 19:42:52 INFO client.RemoteDriver: Connecting to: rhes564:51642
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl: Exception
> in thread "main" java.lang.NoSuchFieldError:
> SPARK_RPC_CLIENT_CONNECT_TIMEOUT
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> org.apache.hive.spark.client.rpc.RpcConfiguration.<clinit>(RpcConfiguration.java:46)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:146)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> java.lang.reflect.Method.invoke(Method.java:606)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>
> 15/12/03 19:42:52 [stderr-redir-1]: INFO client.SparkClientImpl:        at
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>
>
>
>
>
>
>
> Mich Talebzadeh
>
>
>
> Sybase ASE 15 Gold Medal Award 2008
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
>
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15",
> ISBN 978-0-9563693-0-7.
>
> co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4
>
> Publications due shortly:
>
> Complex Event Processing in Heterogeneous Environments, ISBN:
> 978-0-9563693-3-8
>
> Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume
> one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus free,
> therefore neither Peridale Ltd, its subsidiaries nor their employees accept
> any responsibility.
>
>
>
>
>
> -----Original Message-----
> From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
> Sent: 03 December 2015 19:02
> To: 'Marcelo Vanzin' <vanzin@cloudera.com>
> Cc: user@hive.apache.org; 'user' <user@spark.apache.org>
> Subject: RE: Any clue on this error, Exception in thread "main"
> java.lang.NoSuchFieldError: SPARK_RPC_CLIENT_CONNECT_TIMEOUT
>
>
>
> Hi Marcelo.
>
>
>
> So this is the approach I am going to take:
>
>
>
> Use spark 1.3 pre-built
>
> Use Hive 1.2.1. Do not copy over anything to add to hive libraries from
> spark 1.3 libraries Use Hadoop 2.6
>
>
>
> There is no need to mess around with the libraries. I will try to unset my
> CLASSPATH and reset again and try again
>
>
>
>
>
> Thanks,
>
>
>
>
>
> Mich Talebzadeh
>
>
>
> Sybase ASE 15 Gold Medal Award 2008
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15",
> ISBN 978-0-9563693-0-7.
>
> co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4 Publications due shortly:
>
> Complex Event Processing in Heterogeneous Environments, ISBN:
> 978-0-9563693-3-8 Oracle and Sybase, Concepts and Contrasts, ISBN:
> 978-0-9563693-1-4, volume one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus free,
> therefore neither Peridale Ltd, its subsidiaries nor their employees accept
> any responsibility.
>
>
>
> -----Original Message-----
>
> From: Marcelo Vanzin [mailto:vanzin@cloudera.com]
>
> Sent: 03 December 2015 18:45
>
> To: Mich Talebzadeh <mich@peridale.co.uk>
>
> Cc: user@hive.apache.org; user <user@spark.apache.org>
>
> Subject: Re: Any clue on this error, Exception in thread "main"
> java.lang.NoSuchFieldError: SPARK_RPC_CLIENT_CONNECT_TIMEOUT
>
>
>
> On Thu, Dec 3, 2015 at 10:32 AM, Mich Talebzadeh <mich@peridale.co.uk>
> wrote:
>
>
>
>> hduser@rhes564::/usr/lib/spark/logs> hive --version
>
>> SLF4J: Found binding in
>
>> [jar:file:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar!/org
>
>> /slf4j/impl/StaticLoggerBinder.class]
>
>
>
> As I suggested before, you have Spark's assembly in the Hive classpath.
> That's not the way to configure hive-on-spark; if the documentation you're
> following tells you to do that, it's wrong.
>
>
>
> (And sorry Ted, but please ignore Ted's suggestion. Hive-on-Spark should
> work fine with Spark 1.3 if it's configured correctly. You really don't want
> to be overriding Hive classes with the ones shipped in the Spark assembly,
> regardless of the version of Spark being used.)
>
>
>
> --
>
> Marcelo
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
> commands, e-mail: user-help@spark.apache.org



-- 
Marcelo

Mime
View raw message