spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Ahern (JIRA)" <>
Subject [jira] [Created] (SPARK-26314) support Confluent encoded Avro in Spark Structured Streaming
Date Sun, 09 Dec 2018 14:08:00 GMT
David Ahern created SPARK-26314:

             Summary: support Confluent encoded Avro in Spark Structured Streaming
                 Key: SPARK-26314
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.4.0
            Reporter: David Ahern

As Avro has now been added as a first class citizen,


please make Confluent encoded avro work out of the box with Spark Structured Streaming

as described in this link, Avro messages on Kafka encoded with confluent serializer also need
to be decoded with confluent.  It would be great if this worked out of the box


here are details on the Confluent encoding


It's been a year since i worked on anything to do with Avro and Spark Structured Streaming,
but i had to take an approach such as this when getting it to work.  This is what i  used
as a reference at that time


Also, here is another link i found that someone has done in the meantime



This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message