avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-991) Allow combining multiple Avro files within a stream. (no files on disk)
Date Fri, 13 Jan 2012 20:14:42 GMT

    [ https://issues.apache.org/jira/browse/AVRO-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185815#comment-13185815

Tom White commented on AVRO-991:

Is it worth considering expanding the spec to support concatenated Avro data files, just like
gzip allows (http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage )? Then you'd
be able to do

cat file1.avro file2.avro > combined-file.avro

and read combined-file.avro as a regular Avro file.

However, I'm not sure how splitting would work since the sync markers would be different in
the original Avro files. (If only Avro files had a fixed sync marker! Like bzip2 does. This
also tripped me up recently for another reason: two Avro files with the same contents do not
have the same binary representation, so you can't just checksum them to see if they are the
> Allow combining multiple Avro files within a stream. (no files on disk)
> -----------------------------------------------------------------------
>                 Key: AVRO-991
>                 URL: https://issues.apache.org/jira/browse/AVRO-991
>             Project: Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.6.1
>            Reporter: Frank Grimes
> It would be nice to be able to do as follows:
>   cat file1.avro file2.avro | java -jar avro-tools.jar streamcombine > combined-file.avro
> or similarly
>   hadoop dfs -cat hdfs://hadoop/file1.avro hdfs://hadoop/file2.avro | java -jar avro-tools.jar
streamcombine | hdfs -put - hdfs://hadoop/combined-file.avro
> See the following thread for details: http://mail-archives.apache.org/mod_mbox/avro-user/201201.mbox/%3cC08F1DE9-97A8-4D28-B0AD-5E4A7F32F028@gmail.com%3e

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message