Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
MIME-Version: 1.0
In-Reply-To: <CAOt3q9coA0i3hnGY4j=YPozrg3EYJfQ41goA5VhM_xn5-bqvRA@mail.gmail.com>
References: <CABmO9D_Rcxi_DOpuCap5D2jKdsHV2byNeKxs5dw5CiYg=-7Q8A@mail.gmail.com>
 <CAOt3q9coA0i3hnGY4j=YPozrg3EYJfQ41goA5VhM_xn5-bqvRA@mail.gmail.com>
From: Josh <jofo90@gmail.com>
Date: Tue, 15 Nov 2016 20:22:10 +0000
Message-ID: <CABmO9D82RWiWFRpeu+ShVh7PNUz9yOg7V-ckJKMBm-cyjwuXrw@mail.gmail.com>
Subject: Re: Alternative to Avro container files for long-term Avro storage
To: user@avro.apache.org
Content-Type: multipart/alternative; boundary=001a1135058018f1b505415cb669
archived-at: Tue, 15 Nov 2016 20:22:17 -0000

--001a1135058018f1b505415cb669
Content-Type: text/plain; charset=UTF-8

Thanks for the replies,
Originally I wanted to have a Kafka topic with multiple schema types, but
Ken's approach sounds like it could work well so I will try out the single
schema approach with a big union type at the root of the schema.

Josh

On Tue, Nov 15, 2016 at 4:44 PM, John McClean <jmcclean@gmail.com> wrote:

> One approach is to have separate Kafka topics per schema, which evolve
> with use of a schema registry: https://github.com/
> confluentinc/schema-registry. You'd write to the topic with the schema id
> in metadata. You'd write normal avro storage files, knowing when to split
> them based on the changing schema id in the kafka message.
>
> On Tue, Nov 15, 2016 at 2:24 AM, Josh <jofo90@gmail.com> wrote:
>
>> Hi all,
>>
>> I am using a typical Avro->Kafka solution where data is serialized to
>> Avro before it gets written to Kafka and each message is prepended with a
>> schema ID which can be looked up in my schema repository.
>>
>> Now, I want to store the data in long-term storage by writing data from
>> Kafka->S3.
>>
>> I know that the usual way to store Avro in storage is using Avro
>> container files, however a container file can only contain messages encoded
>> with a single Avro schema. In my case, the messages may be encoded with
>> difference schemas, and I need to retain the order of the messages (so that
>> they can be replayed into Kafka, in order). Therefore, a single file in S3
>> needs to contain messages encoded with different schemas and so I can't use
>> Avro container files.
>>
>> I was wondering what would be a good solution to this? What format could
>> I use to store my Avro data, such that a single data file can contain
>> messages encoded with different schemas? Should I store the messages with a
>> prepended schema ID, similar to what I do in Kafka? In that case, how could
>> I separate the messages in the file?
>>
>> Thanks for any advice,
>> Josh
>>
>
>

--001a1135058018f1b505415cb669
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks for the replies,<br><div>Originally I wanted to hav=
e a Kafka topic with multiple schema types, but Ken&#39;s approach sounds l=
ike it could work well so I will try out the single schema approach with a =
big union type at the root of the schema.</div><div><br></div><div>Josh</di=
v></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, N=
ov 15, 2016 at 4:44 PM, John McClean <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:jmcclean@gmail.com" target=3D"_blank">jmcclean@gmail.com</a>&gt;</span> w=
rote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;borde=
r-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">One approach is to=
 have separate Kafka topics per schema, which evolve with use of a schema r=
egistry:=C2=A0<a href=3D"https://github.com/confluentinc/schema-registry" t=
arget=3D"_blank">https://github.com/<wbr>confluentinc/schema-registry</a>. =
You&#39;d write to the topic with the schema id in metadata. You&#39;d writ=
e normal avro storage files, knowing when to split them based on the changi=
ng schema id in the kafka message.</div><div class=3D"HOEnZb"><div class=3D=
"h5"><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Nov =
15, 2016 at 2:24 AM, Josh <span dir=3D"ltr">&lt;<a href=3D"mailto:jofo90@gm=
ail.com" target=3D"_blank">jofo90@gmail.com</a>&gt;</span> wrote:<br><block=
quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
 solid;padding-left:1ex"><div dir=3D"ltr">Hi all,<div><br></div><div>I am u=
sing a typical Avro-&gt;Kafka solution where data is serialized to Avro bef=
ore it gets written to Kafka and each message is prepended with a schema ID=
 which can be looked up in my schema repository.</div><div><br></div><div>N=
ow, I want to store the data in long-term storage by writing data from Kafk=
a-&gt;S3.=C2=A0</div><div><br></div><div>I know that the usual way to store=
 Avro in storage is using Avro container files, however a container file ca=
n only contain messages encoded with a single Avro schema. In my case, the =
messages may be encoded with difference schemas, and I need to retain the o=
rder of the messages (so that they can be replayed into Kafka, in order). T=
herefore, a single file in S3 needs to contain messages encoded with differ=
ent schemas and so I can&#39;t use Avro container files.=C2=A0</div><div><b=
r></div><div>I was wondering what would be a good solution to this? What fo=
rmat could I use to store my Avro data, such that a single data file can co=
ntain messages encoded with different schemas? Should I store the messages =
with a prepended schema ID, similar to what I do in Kafka? In that case, ho=
w could I separate the messages in the file?</div><div><br></div><div>Thank=
s for any advice,</div><div>Josh</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a1135058018f1b505415cb669--