avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yang Yang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AVRO-127) Avro should support multiple schemas from the same AVRO file
Date Mon, 28 Sep 2009 20:41:16 GMT

    [ https://issues.apache.org/jira/browse/AVRO-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760376#action_12760376

Yang Yang commented on AVRO-127:

yes, this is exactly what we want.

the schema change frequency is fine, we will have about at most 1--2 changes each month, so
about 1 or 2 daily files out of every 30 files will have
more than 1 schema in them. 

having a new constructor isn't so urgently needed, the above "empty union--->addSchema()
"  approach is good enough.

but there is indeed a problem: I normally use GenericDatumWriter so that  I can use it directly
on HDFS , instead of facing a file (the hadoop FileInput/FileOutputFormat 
API gives me a DataInputStream to work with, not a file), it seems that GenericDatumWriter
does not have addSchema() method. does that need to be copied over 
to GenericDatumWriter ?


>  Avro should support multiple schemas from the same AVRO file 
> --------------------------------------------------------------
>                 Key: AVRO-127
>                 URL: https://issues.apache.org/jira/browse/AVRO-127
>             Project: Avro
>          Issue Type: New Feature
>         Environment: all systems
>            Reporter: Yang Yang
> in our application, we often have to merge together all the day's data into one daily
> data file schemas must be changing within the day, so we have different schemas within
the same file.
> AVRO should support multiple schemas within the same file, ideally different schemas
can be parsed out
> to different children classes of the same schema class.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message