avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Haviv <daniel.ha...@veracity-group.com>
Subject Parsing avro binary data from Spark Streaming
Date Fri, 25 Sep 2015 16:18:43 GMT
Hi,
I'm receiving avro data from Kafka in my Spark Streaming app.
When reading the data directly from disk I would have just used the
following manner to parse it :
val avroRDD = sc.hadoopFile[AvroWrapper[GenericRecord], NullWritable,
AvroInputFormat[GenericRecord]]("/incoming_1k").coalesce(10)
val txtRDD = avroRDD.map(l => {l._1.datum.toString} )

I would like to do the same with avro data coming in from kafka, so I'm
doing the following:
val avroStream = KafkaUtils.createDirectStream[Array[Byte], Array[Byte],
DefaultDecoder, DefaultDecoder](ssc, kafkaParams, topicSet)

This leaves me with a byte array and I can't find any example on how to
convert a byte array to either a GenericRecord or to my avro class.

Any help will be appreciated.

Daniel

Mime
View raw message