avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakub Kahovec (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AVRO-1827) Handling correctly optional fields when converting Protobuf to Avro
Date Mon, 11 Apr 2016 14:47:25 GMT
Jakub Kahovec created AVRO-1827:
-----------------------------------

             Summary: Handling correctly optional fields when converting Protobuf to Avro
                 Key: AVRO-1827
                 URL: https://issues.apache.org/jira/browse/AVRO-1827
             Project: Avro
          Issue Type: Improvement
    Affects Versions: 1.8.0, 1.7.7
            Reporter: Jakub Kahovec


Hello,
as of the current implementation of converting protobuf files into avro format, protobuf optional
fields are being  given default values in the avro schema if not specified explicitly. 
So for instance when the protobuf field is defined as  
{quote}
optional int64 fieldInt64 = 1;
{quote}
in the avro schema it appears as
{quote}
 "name" : "fieldInt64",
  "type" : "long",
  "default" : 0
{quote}
The problem with this implementation is that we are losing information about whether the field
was present or not in the original protobuf, as when we ask for this field's value in avro
we will be given the default value. 

What I'm proposing instead is that if the field in the protobuf is defined as optional and
has no default value then the generated avro schema type will us a union comprising the matching
type and null type with default value null. It is going to look like this:
{quote}
 "name" : "fieldIn64",
  "type" : [ "null", "long" ],
  "default" : null
{quote}
I'm aware that is a breaking change but I think that is the proper way how to handle optional
fields.

I've also  created a patch which fixes the conversion

Jakub 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message