avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1873) avro gem doesn't compatible with other languages with snappy compression
Date Mon, 12 Sep 2016 16:09:21 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15484518#comment-15484518

ASF subversion and git services commented on AVRO-1873:

Commit 832512edcd7591c238c35b5a479e15ac0709e4cb in avro's branch refs/heads/master from [~rdblue]
[ https://git-wip-us.apache.org/repos/asf?p=avro.git;h=832512e ]

AVRO-1873: Add CRC32 checksum to Snappy-compressed blocks.

Java and other implementations require this CRC32 checksum of the
uncompressed content in order to read the data. This implements the
checksum, with backward-compatibility for files written by old versions
of avro-ruby. If the checksum doesn't match or if decompression fails
with the last 4 bytes removed as the checksum, avro-ruby will decompress
the incoming bytes and pass them on assuming that the file is from an
old reader.

Closes #121.

> avro gem doesn't compatible with other languages with snappy compression
> ------------------------------------------------------------------------
>                 Key: AVRO-1873
>                 URL: https://issues.apache.org/jira/browse/AVRO-1873
>             Project: Avro
>          Issue Type: Bug
>          Components: ruby
>    Affects Versions: 1.8.1
>         Environment: CentOS 6.8 64bit, Snappy 1.1.0, Python 3.5, Ruby 2.2.3
>            Reporter: Pumsuk Cho
>            Assignee: Ryan Blue
>            Priority: Blocker
>             Fix For: 1.8.2
> I've tested avro gem today, then found some weird result.
> With python library like "fastavro", generated an avro file snappy compressed. This file
works fine with avro-tools-1.8.1.jar.
> java -jar avro-tools-1.8.1.jar tojson testing.avro returns what I expected.
> But NOT compatible with ruby using avro gem returns "Invalid Input" message. And snappy
compressed avro file made with avro gem doesn't work with avro-tools nor in python with avro-python3
and fastavro.
> my ruby codes are below:
> schema = Avro::Schema.paese(File.open('test.avsc', 'r').read)
> avrofile = File.open('test.avro', 'wb')
> writer = Avro::IO::DatumWriter.new(schema)
> datawriter = Avro::DataFile::Writer.new file, writer, schema, 'snappy'
> datawriter<< {"title" => "Avro", "author" => "Apache Foundation"}
> datawriter.close

This message was sent by Atlassian JIRA

View raw message