spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tcg <Gullo_Tho...@bah.com>
Subject Reading Avro Sequence Files
Date Fri, 18 Jul 2014 20:47:10 GMT
I'm trying to read and an Avro Sequence File using the sequenceFile method on
the spark context object and I get a NullPointerException.  If I read the
file outside of Spark using AvroSequenceFile.Reader I don't have any
problems.

Anyone have success in doing this?

Below is one I typed and saw at the spark shell:

scala>var myAvroSequenceFile = sc.sequenceFile("hdfs://<my url is here>",
classOf[AvroKey[GenericRecord], ClassOf[AvroValue[GenericRecord]])

scala>myAvroSequenceFile.first
14/07/18 16:31:31 INFO FileInputFormat: Total input paths to process : 1
14/07/18 16:31:31 INFO SparkContext: Starting job: first at <console>:18
14/07/18 16:31:31 INFO DAGScheduler: Got job 2 (first at <console>:18) with
1 output partitions (allowLocal=true)
14/07/18 16:31:31 INFO DAGScheduler: Final stage: Stage 2(first at
<console>:18)
14/07/18 16:31:31 INFO DAGScheduler: Parents of final stage: List()
14/07/18 16:31:31 INFO DAGScheduler: Missing parents: List()
14/07/18 16:31:31 INFO DAGScheduler: Computing the requested partition
locally
14/07/18 16:31:31 INFO HadoopRDD: Input split: hdfs://<my url>
14/07/18 16:31:31 INFO DAGScheduler: Failed to run first at <console>:18
java.lang.NullPointerException
        at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1902)
        at
org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
        at
org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
        at
org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
        at
org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:190)
        at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:181)
        at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:93)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        at
org.apache.spark.scheduler.DAGScheduler.runLocallyWithinThread(DAGScheduler.scala:574)
        at
org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:559)



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Reading-Avro-Sequence-Files-tp10201.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message