spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul Swaminathan <rahul.swaminat...@duke.edu>
Subject WordCount fails in .textFile() method
Date Tue, 02 Dec 2014 21:17:43 GMT
Hi,

I am trying to run JavaWordCount without using the spark-submit script. I have copied the
source code for JavaWordCount and am using a JavaSparkContext with the following:

SparkConf conf = new SparkConf().setAppName("JavaWordCount");
conf.set("spark.io.compression.codec","org.apache.spark.io.LZ4CompressionCodec");
conf.setMaster("spark://127.0.0.1:7077").setSparkHome(System.getenv("SPARK_HOME"));
JavaSparkContext jsc = new JavaSparkContext(conf);
jsc.addJar("myJar.jar");
new JavaWordCount(jsc).doJob();

I am getting the following error in the .textFile() method:


Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.hash.HashFunction.hashInt(I)Lcom/google/common/hash/HashCode;

at org.apache.spark.util.collection.OpenHashSet.org$apache$spark$util$collection$OpenHashSet$$hashcode(OpenHashSet.scala:261)


What can I do to solve this issue? It works fine when running from command line with spark-submit
script.


Thanks,


Rahul



Mime
View raw message