pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [pulsar] sijie commented on issue #3741: POJO AvroSchema always allowNull
Date Tue, 05 Mar 2019 06:58:04 GMT
sijie commented on issue #3741: POJO AvroSchema always allowNull
URL: https://github.com/apache/pulsar/issues/3741#issuecomment-469563201
 
 
   @jerrypeng  correct. I added one more to complete the picture.
   
   ```
           log.info("Pulsar AllowNull                  :{}", new String(AvroSchema.of(User.class).getSchemaInfo().getSchema()));
           log.info("Pulsar shaded AVRO NOT AllowNull  :{}", org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.get().getSchema(User.class));
           log.info("Avro AllowNull                    :{}", ReflectData.AllowNull.get().getSchema(User.class));
           log.info("Avro Not AllowNull                :{}", ReflectData.get().getSchema(User.class));
   ```
   
   ```
   Pulsar AllowNull                  :{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":["null","string"],"default":null},{"name":"favorite_number","type":["null","int"],"default":null},{"name":"favorite_color","type":["null","string"],"default":null},{"name":"age","type":"int"}]}
   Pulsar shaded AVRO NOT AllowNull  :{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":"int"},{"name":"favorite_color","type":"string"},{"name":"age","type":"int"}]}
   Avro AllowNull                    :{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":["int","null"]},{"name":"favorite_color","type":"string"},{"name":"age","type":"int","default":19}]}
   Avro Not AllowNull                :{"type":"record","name":"User","namespace":"example.avro","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":["int","null"]},{"name":"favorite_color","type":"string"},{"name":"age","type":"int","default":19}]}
   ```
   
   As you described above, AllowNull doesn't matter if the POJO is generated by AVRO, because
AVRO will try to use `SCHEMA$` field to retrieve the schema. But If the POJO is not generated
by AVRO, AVRO will attempt to parse it and AllowNull will generating two different incompatible
schemas.
   
   my tests avoid covered all the cases.
   
   1) `Avro AllowNull` and `Avro Not AllowNull` are using unshaded avro library to parse avro
generated POJO. in this way, they will always produce the same schema no matter `AllowNull`
used or not. 
   
   2) `Pulsar AllowNull` and `Pulsar shade avro Not Allow Null` are using shaded avro library
to parse avro generated POJO. In this way, they will produce a completely different schema.
`AllowNull` matters here. 
   
   The 2) indicates two problems:
   
   a) for normal POJO (not generated by AVRO), `AllowNull` is producing incompatible schemas.
so we need #3752 
   
   b) for avro generated POJO, shaded AVRO library (the one used by Pulsar) doesn't recognize
it is an AVRO generated POJO. We need a separate fix to make sure Pulsar shaded AVRO working
with avro generated POJO.
   
   
   
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message