pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] sijie opened a new pull request #3670: [schema] Introduce multi version generic record schema
Date Fri, 22 Feb 2019 09:37:45 GMT
sijie opened a new pull request #3670: [schema] Introduce multi version generic record schema
URL: https://github.com/apache/pulsar/pull/3670
 
 
   *Motivation*
   
   Currently AUTO_CONSUME only supports decoding records from latest schema.
   All the schema versions are lost. It makes AUTO_CONSUME less useful in some use cases,
   such as CDC. Because there is no way for the applications to know which version of schema
   that a message is using.
   
   In order to support multi-version schema, we need to propagate schema version from
   message header through schema#decode method to the decoded record.
   
   *Modifications*
   
   - Introduce a new decode method `decode(byte[] data, byte[] schemaVersion)`. This allows
the implementation
     to leverage the schema version.
   - Introduce a method `supportSchemaVersioning` to tell which decode methods to use. Because
most of the schema
     implementations such as primitive schemas and POJO based schema doesn't make any sense
to use schema version.
   - Introduce a SchemaProvider which returns a specific schema instance for a given schema
version
   - Implement a MultiVersionGenericRecordSchema which decode the messages based on schema
version. All the records
     decoded by this schema will have schema version and its corresponding schema definitions.
   
   *NOTES*
   
   This implementation only introduce the mechanism. But it doesn't wire the multi-version
schema
   with auto_consume schema. There will be a subsequent pull request on implementing a schema
provider
   that fetches and caches schemas from brokers.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message