spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yana Kadiyska <yana.kadiy...@gmail.com>
Subject Re: Can Spark1.4 work with CDH4.6
Date Wed, 24 Jun 2015 19:38:44 GMT
Thanks, that did seem to make a difference. I am a bit scared of this
approach as spark itself has a different guava dependency but the error
does go away this way

On Wed, Jun 24, 2015 at 10:04 AM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> Can you try to add those jars in the SPARK_CLASSPATH and give it a try?
>
> Thanks
> Best Regards
>
> On Wed, Jun 24, 2015 at 12:07 AM, Yana Kadiyska <yana.kadiyska@gmail.com>
> wrote:
>
>> Hi folks, I have been using Spark against an external Metastore service
>> which runs Hive with Cdh 4.6
>>
>> In Spark 1.2, I was able to successfully connect by building with the
>> following:
>>
>> ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0
>> -Phive-thriftserver -Phive-0.12.0
>>
>> I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor of
>> spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars
>>
>> When I tried to use this setup spark-shell fails for me with the
>> following error:
>>
>> 15/06/23 18:18:07 INFO hive.HiveContext: Initializing HiveMetastoreConnection version
0.12.0 using [Ljava.net.URL;@7b7a9a6c
>> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
when creating Hive client using classpath: file:/hive/lib/guava-11.0.2.jar, file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar,
file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar,
file:/hive/lib/commons-logging-1.0.4.jar
>>
>> ‚Äč
>>
>> I don't know why it's not seeing the class -- it's in the guava jar. If
>> anyone has had success with 0.12 version please let me know what jars need
>> to be on the classpath. I think my Hive version might be too outdated but I
>> don't control the metastore and I had success with Spark1.2 so I'm hoping...
>>
>>
>>
>

Mime
View raw message