avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <scottca...@apache.org>
Subject Re: Avro 1.6.1 BinaryEncoder.writeBytes doesn't work on OS X
Date Wed, 08 Feb 2012 17:50:13 GMT
Yes, it appears as though GenericDatumWriter does not support native
ByteBuffers.  FileChannel returns native ByteBuffers, not HeapByteBuffer.

Some options:
* Read the file contents as a byte[], and wrap a ByteBuffer around that
* Create a new heap ByteBuffer and copy the contents of the file channel's
byte buffer into it.

Please file a bug in JIRA,

Thanks!

On 2/8/12 6:52 AM, "karthik ramachandran" <kramachandran@gmail.com> wrote:

> Hi all, 
> 
> I'm running into a problem with BinaryEncoder.writeBytes.  I'm trying to wrap
> some documents in an Avro structure and then write them into an Avro contain
> file. 
> 
> My code looks roughly like this :
> 
>         File file = new File(...);
>         ClassLoader cl = SimpleAvroExample.class.
> getClassLoader();
>         InputStream isSimpleWrapperSchema=
> cl.getResourceAsStream("simple_wrapper.avpr");
>         
>         Schema.Parser parser = new Schema.Parser();
>         Schema schemaSimpleWrapper = parser.parse(isSimpleWrapperSchema);
>         GenericDatumWriter<GenericRecord> gdw = new
> GenericDatumWriter<GenericRecord>(schemaSimpleWrapper);
>         DataFileWriter writer = new DataFileWriter(gdw);
> 
>         File outputFile = new File(...);
>         outputFile.createNewFile();
>         
>         writer.create(schemaSimpleWrapper, outputFile);
>         
>         
>         for(File documentFile : file.listFiles())
>         {
>         
>             GenericRecord r =  new GenericData.Record(schemaSimpleWrapper);
>             FileInputStream fis = new FileInputStream(documentFile);
>             FileChannel channel = fis.getChannel();
>             
>             r.put("filename", documentFile.getName());
>         
>              //Copying the whole file.
>             ByteBuffer byteBuffer = channel.map(FileChannel.MapMode.READ_ONLY,
> 0, documentFile.length());
>             r.put("body", byteBuffer);
>             
>             
>           
>             
>             writer.append(r);
>      
>             channel.close();
>             fis.close();
>             
>         }
> 
> This code exceptions out :
>       Exception in thread "main"
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.UnsupportedOperationException
>         at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:261)
>         at org.iqt.cdl.SimpleAvroExample.main(SimpleAvroExample.java:107)
>      Caused by: java.lang.UnsupportedOperationException
>         at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968)
>        at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61)
>        at 
> org.apache.avro.generic.GenericDatumWriter.writeBytes(GenericDatumWriter.java:
> 199)
>        at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:76)
>        at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java
> :105)
>        at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
>        at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
>        at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:255)
>        ... 1 more
> 
> 
> The problem seems to be that the writeBytes function on line 61 does the
> following : 
>      int start = bytes.arrayOffset() + pos.
> 
> Where byes is a java.nio.ByteBuffer.
> 
> According to the JavaDocs for ByteBuffer the arrayOffset fuctional is
> optional.  
> 
> As near as I can tell this OS X implementation of ByteBuffer does not support
> his functionality.
> 
> 
> -- 
> Karthik Ramachandran
> Mobile: 412-606-8981



Mime
View raw message