avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Grey <mgthesec...@spotify.com.INVALID>
Subject Custom logical type associations
Date Tue, 20 Jun 2017 15:05:31 GMT
Hello user, dev,

Recently we've been exploring the possibility of migrating our wire
serialization format to Avro.  This is intended to better integrate with
our data-driven systems in Hadoop, Google Cloud Dataflow and elsewhere.  We
use avro pretty widely for all our data "at rest" in storage in HDFS and in
the cloud, so it seemed like a good approach to experiment with.

Generally we store our avro schemas in dedicated repositories separate from
application code and then use the SpecificRecord classes generated by the
maven-avro-compiler plugin in our applications.  This has worked well for
us in batch processing settings and we'd like to try it at the perimeter of
our REST apis.

I recently discovered the addition of logical types to avro, which could be
useful specifically for our date fields, which we presently represent as
ISO8601 strings.  However, it appears the new type is a decoration on an
int.  This leads me to several questions:

1.  There is some discussion in the issue tracker
<https://issues.apache.org/jira/browse/AVRO-739> regarding the use of
ISO8601 strings vs integer days since epoch.  There was mention of adding a
compiler flag to switch between the two.  Is this still desirable?  It is
unclear what the conclusion was from the discussion.

2.  Is it possible for us to customize the types and logicalTypes
associated with a given type, either through using the Reflection based API
or through extending encoders and decoders to better suit our needs?  There
appears to be prior art of this nature extending the JSON serdes to use the
natural rest shape
<https://issues.apache.org/jira/browse/AVRO-1582?focusedCommentId=15513391&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15513391>
(I
should mention this patch is also something we find desirable).  Our main
goal here would be to make our string ISO8601 fields backward compatible
but return jdk8 LocalDates.

Any help or pointers would be appreciated.  Our team has a high interest in
this and would be willing to commit to long term maintenance and review of
any patches.

~MGII

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message