avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Fuller (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AVRO-1081) GenericDatumWriter does not support native ByteBuffers
Date Mon, 14 May 2012 12:12:03 GMT

     [ https://issues.apache.org/jira/browse/AVRO-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Fuller updated AVRO-1081:
--------------------------------

    Attachment: patch.diff.txt
                ByteBufferTest.java

Attached one possible way of fixing this.

I am not 100% happy with the solution, but it should work for our case for now. We are writing
several avro files concurrently from within a heavily multithreaded application, and cannot
afford to load many of the files into memory at once at that point.

When reading the files again (in hadoop m/r job) it's ok for now to read the bytes back to
memory (will read them into a byte array, but for the purpose of the test wanted to put them
into a byte buffer).

Am also now wondering whether you might consider direct support for "File" (or something like
FileData) in avro. This would allow a way to include into avro items which exceed the amount
of available memory. But maybe it's not a requirement anybody else would have...
                
> GenericDatumWriter does not support native ByteBuffers
> ------------------------------------------------------
>
>                 Key: AVRO-1081
>                 URL: https://issues.apache.org/jira/browse/AVRO-1081
>             Project: Avro
>          Issue Type: Bug
>    Affects Versions: 1.6.3
>            Reporter: Robert Fuller
>         Attachments: ByteBufferTest.java, ByteBufferTest.java, patch.diff.txt, patch.diff.txt
>
>
> An exception is thrown when trying to encode bytes backed by a file.
> java.lang.UnsupportedOperationException: null
> 	at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968) ~[na:1.6.0_31]
> 	at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61) ~[avro-1.6.3.jar:1.6.3]
> Note arrayOffset is an optional method, see:
> http://docs.oracle.com/javase/6/docs/api/java/nio/ByteBuffer.html#arrayOffset%28%29
> FileChannel returns native ByteBuffer not HeapedByteBuffer
> See here:
> http://mail-archives.apache.org/mod_mbox/avro-user/201202.mbox/%3CCB57F421.6BFE2%25scott@richrelevance.com%3E

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message