hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Furcy Pin <furcy....@flaminem.com>
Subject Re: Hive on Spark not working
Date Tue, 29 Nov 2016 07:56:25 GMT
ClassNotFoundException generally means that jars are missing from your
class path.

You probably need to link the spark jar to $HIVE_HOME/lib
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive

On Tue, Nov 29, 2016 at 2:03 AM, Joaquin Alzola <Joaquin.Alzola@lebara.com>
wrote:

> Hi Guys
>
>
>
> No matter what I do that when I execute “select count(*) from employee” I
> get the following output on the logs:
>
> It is quiet funny because if I put hive.execution.engine=mr the output is
> correct. If I put hive.execution.engine=spark then I get the bellow errors.
>
> If I do the search directly through spark-shell it work great.
>
> +-------+
>
> |    _c0|
>
> +-------+
>
> |1005635|
>
> +-------+
>
> So there has to be a problem from hive to spark.
>
>
>
> Seems as the RPC(??) connection is not setup …. Can somebody guide me on
> what to look for.
>
> spark.master=spark://172.16.173.31:7077
>
> hive.execution.engine=spark
>
> spark.executor.extraClassPath    /mnt/spark/lib/spark-1.6.2-
> yarn-shuffle.jar:/mnt/hive/lib/hive-exec-2.0.1.jar
>
>
>
> Hive2.0.1à Spark 1.6.2 –> Hadoop – 2.6.5 à Scala 2.10
>
>
>
> 2016-11-29T00:35:11,099 WARN  [RPC-Handler-2]: rpc.RpcDispatcher
> (RpcDispatcher.java:handleError(142)) - Received error
> message:io.netty.handler.codec.DecoderException: java.lang.NoClassDefFoundError:
> org/apache/hive/spark/client/Job
>
>         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(
> ByteToMessageDecoder.java:358)
>
>         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(
> ByteToMessageDecoder.java:230)
>
>         at io.netty.handler.codec.ByteToMessageCodec.channelRead(
> ByteToMessageCodec.java:103)
>
>         at io.netty.channel.AbstractChannelHandlerContext.
> invokeChannelRead(AbstractChannelHandlerContext.java:308)
>
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:294)
>
>         at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(
> ChannelInboundHandlerAdapter.java:86)
>
>         at io.netty.channel.AbstractChannelHandlerContext.
> invokeChannelRead(AbstractChannelHandlerContext.java:308)
>
>         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(
> AbstractChannelHandlerContext.java:294)
>
>         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(
> DefaultChannelPipeline.java:846)
>
>         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(
> AbstractNioByteChannel.java:131)
>
>         at io.netty.channel.nio.NioEventLoop.processSelectedKey(
> NioEventLoop.java:511)
>
>         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
> NioEventLoop.java:468)
>
>         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(
> NioEventLoop.java:382)
>
>         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>
>         at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:111)
>
>         at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.lang.NoClassDefFoundError: org/apache/hive/spark/client/
> Job
>
>         at java.lang.ClassLoader.defineClass1(Native Method)
>
>         at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>
>         at java.security.SecureClassLoader.defineClass(
> SecureClassLoader.java:142)
>
>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>
>         at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>         at java.lang.Class.forName0(Native Method)
>
>         at java.lang.Class.forName(Class.java:348)
>
>         at org.apache.hive.com.esotericsoftware.kryo.util.
> DefaultClassResolver.readName(DefaultClassResolver.java:154)
>
>         at org.apache.hive.com.esotericsoftware.kryo.util.
> DefaultClassResolver.readClass(DefaultClassResolver.java:133)
>
>         at org.apache.hive.com.esotericsoftware.kryo.Kryo.
> readClass(Kryo.java:670)
>
>         at org.apache.hive.com.esotericsoftware.kryo.
> serializers.ObjectField.read(ObjectField.java:118)
>
>         at org.apache.hive.com.esotericsoftware.kryo.
> serializers.FieldSerializer.read(FieldSerializer.java:551)
>
>         at org.apache.hive.com.esotericsoftware.kryo.Kryo.
> readClassAndObject(Kryo.java:790)
>
>         at org.apache.hive.spark.client.rpc.KryoMessageCodec.decode(
> KryoMessageCodec.java:97)
>
>         at io.netty.handler.codec.ByteToMessageCodec$1.decode(
> ByteToMessageCodec.java:42)
>
>         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(
> ByteToMessageDecoder.java:327)
>
>         ... 15 more
>
> Caused by: java.lang.ClassNotFoundException: org.apache.hive.spark.client.
> Job
>
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>         ... 39 more
> This email is confidential and may be subject to privilege. If you are not
> the intended recipient, please do not copy or disclose its content but
> contact the sender immediately upon receipt.
>

Mime
View raw message