hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mich Talebzadeh" <m...@peridale.co.uk>
Subject Building spark 1.3 from source code to work with Hive 1.2.1
Date Thu, 03 Dec 2015 10:28:55 GMT
Hi,

 

I have seen mails that state that the user has managed to build spark 1.3 to
work with Hive. I tried Spark 1.5.2 but no luck

 

I downloaded spark source 1.3 source code spark-1.3.0.tar and built it as
follows

 

./make-distribution.sh --name "hadoop2-without-hive" --tgz
"-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"

 

This successfully completed and created the tarred zip file. I then created
spark 1.3 tree from this zipped file. $SPARK_HOME is /usr/lib/spark

 

Other steps that I performed:

 

1.    In $HIVE_HOME/lib , I copied  spark-assembly-1.3.0-hadoop2.4.0.jar  to
this directory

2.  In $SPARK_HOME/conf I created a syblink to
/usr/lib/hive/conf/hive-site.xml

 

Then I tried to start spark master node

 

/usr/lib/spark/sbin/start-master.sh

 

I get the following error:

 

 

cat
/usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Mast
er-1-rhes564.out

Spark Command: /usr/java/latest/bin/java -cp
:/usr/lib/spark/sbin/../conf:/usr/lib/spark/lib/spark-assembly-1.3.0-hadoop2
.4.0.jar:/home/hduser/hadoop-2.6.0/etc/hadoop -XX:MaxPermSize=128m
-Dspark.akka.logLifecycleEvents=true -Xms512m -Xmx512m
org.apache.spark.deploy.master.Master --ip rhes564 --port 7077 --webui-port
8080

========================================

 

Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger

        at java.lang.Class.getDeclaredMethods0(Native Method)

        at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)

        at java.lang.Class.getMethod0(Class.java:2764)

        at java.lang.Class.getMethod(Class.java:1653)

        at
sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)

        at
sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)

Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger

        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

 

I also notice that in /usr/lib/spark/lib, I only have the following jar
files

 

-rw-r--r-- 1 hduser hadoop 98795479 Dec  3 09:03
spark-examples-1.3.0-hadoop2.4.0.jar

-rw-r--r-- 1 hduser hadoop 98187168 Dec  3 09:03
spark-assembly-1.3.0-hadoop2.4.0.jar

-rw-r--r-- 1 hduser hadoop  4136760 Dec  3 09:03
spark-1.3.0-yarn-shuffle.jar

 

Wheras in pre-build downloaded one --> /usr/lib/spark-1.3.0-bin-hadoop2.4,
there are additional  JAR files

 

-rw-rw-r-- 1 hduser hadoop   1890075 Mar  6  2015
datanucleus-core-3.2.10.jar

-rw-rw-r-- 1 hduser hadoop 112446389 Mar  6  2015
spark-examples-1.3.0-hadoop2.4.0.jar

-rw-rw-r-- 1 hduser hadoop 159319006 Mar  6  2015
spark-assembly-1.3.0-hadoop2.4.0.jar

-rw-rw-r-- 1 hduser hadoop   4136744 Mar  6  2015
spark-1.3.0-yarn-shuffle.jar

-rw-rw-r-- 1 hduser hadoop   1809447 Mar  6  2015
datanucleus-rdbms-3.2.9.jar

-rw-rw-r-- 1 hduser hadoop    339666 Mar  6  2015
datanucleus-api-jdo-3.2.6.jar

 

Any ideas what is is missing? I am sure someone has sorted this one out
before.

 

 

Thanks,

 

Mich

 

 

 

NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Technology
Ltd, its subsidiaries or their employees, unless expressly so stated. It is
the responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.

 


Mime
View raw message