spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinoth Sankar <>
Subject ClassCastException while reading data from HDFS through Spark
Date Wed, 07 Oct 2015 09:11:09 GMT
 I'm just reading data from HDFS through Spark. It throws
*java.lang.ClassCastException: cannot be cast to* at line no 6. I never used LongWritable
in my code, no idea how the data was in that format.

Note : I'm not using MapReduce Concepts and also I'm not creating Jobs
explicitly. So i can't use job.setMapOutputKeyClass and

JavaPairRDD<IntWritable, BytesWritable> hdfsContent =
sparkContext.sequenceFile(hdfsPath, IntWritable.class, BytesWritable.class);
JavaRDD<FileData> lines = Function<Tuple2<IntWritable,
BytesWritable>, FileData>()
public FileData call(Tuple2<IntWritable, BytesWritable> tuple2) throws
byte[] bytes = tuple2._2().getBytes();
return FileData.parseFrom(bytes);

View raw message