kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Roesler <j...@confluent.io>
Subject Re: [DISCUSS] KIP-466: Add support for List<T> serialization and deserialization
Date Tue, 18 Jun 2019 22:06:46 GMT
Hi Daniyar,

That's a very clever solution!

One observation is that, now, this is what we might call a polymorphic
serde. That is, you're detecting the actual concrete type and then
promising to produce the exact same concrete type on read. There are
some inherent problems with this approach, which in general require
some kind of  schema registry (not necessarily Schema Registry, just
any registry for schemas) to solve.

Notice that every serialized record has quite a bit of duplicated
information: the concrete type as well as a byte to indicate whether
the value type is a fixed size, and, if so, an integer to indicate the
actual size. These constitute a schema, of sorts, because they tell us
later how exactly to deserialize the data. Unfortunately, this
information is completely redundant. In all likelihood, the
information will be exactly the same for every record in the topic.
This problem is essentially the core motivation for serializations
like Avro: to move the schema outside of the serialization itself, so
that the records won't contain so much redundant information.

In this light, I'm wondering if it makes sense to go back to something
like what you had earlier in which you don't support perfectly
preserving the concrete type for _this_ serde, but instead just
support deserializing to _some_ List. Then, you could defer full,
perfect, type preservation to serdes that have an external system in
which to register their type information.

There does exist an alternative, if we really do want to preserve the
concrete type (which does seem kind of nice). You can add a
configuration option specifically for the serde to configure what the
list type will be, and maybe what the element type is, as well.

As far as "related work" goes, you might be interested to take a look
at how Jackson can be configured to deserialize into a specific,
arbitrarily nested, generically parameterized class structure.
Specifically, you might find


On Mon, Jun 17, 2019 at 12:38 PM Development <dev@yeralin.net> wrote:
> bump

View raw message