samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luis Casillas <luis.casil...@progressfin.com>
Subject Re: Sample code or tutorial for writing/reading Avro type message in Samza
Date Thu, 19 Nov 2015 22:22:24 GMT

I did a Samza proof of concept project recently and I ended up writing this code:

https://gist.github.com/ldcasillas-progreso/871af3c1a1790be975fd

In the end, however, I switched the project from Avro to JSON.  The issue is that Avro is
designed to work with its self-describing container file format, which embeds the schema used
to write the records in the file.  Avro’s schema evolution features rely on this embedded
schema; when the embedded schema and the reader’s schema are not equal, Avro uses its special
rules to translate the old data to the new schema.

But when you’re working with Kafka/Samza, there is no container file.  Therefore, none of
the schema evolution tools work.  Therefore, if you change your Avro schema, you likely won’t
be able to read any of the old messages again.

There’s a Kafka Avro schema registry project that aims to fix this:

https://github.com/confluentinc/schema-registry

I tried it but the released version just was not mature enough—which is why I ended up using
JSON.  But I did write a Serde that encodes/decodes the Avro objects in JSON:

https://gist.github.com/ldcasillas-progreso/3611d40d2833aa62c1b3

Hope this helps.





On 11/17/15, 12:32 AM, "Selina Tech" <swucareer99@gmail.com> wrote:

>Dear All:
>     Do you know where I can find the tutorial or sample code for writing
>Avro type message to Kafka and reading Avro type message from Kafka in
>Samza?
>      I am wondering how should I serialized GenericRecord to byte and
>deserialized it?
>     Your comments/suggestion are highly appreciated.
>
>Sincerely,
>Selina


-----------
This message and any files or text attached to it are intended only for the recipients named
above, and contain information that is confidential or privileged. If you are not an intended
recipient, you must not read, copy, use or disclose this communication. Please also notify
the sender by replying to this message, and then delete all copies of it from your system.

Este mensaje y cualquier archivo o texto adjunto es dirigido solamente a los destinatarios
especificados en el encabezado y contiene información confidencial y/o privilegiada. Si usted
no es el destinatario no deberá leer, copiar, usar o divulgar el contenido. Por favor notifique
al remitente, respondiendo a esté mensaje y elimine todas las copias del mismo de su sistema.
Mime
View raw message