avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From karthik ramachandran <kramachand...@gmail.com>
Subject Avro 1.6.1 BinaryEncoder.writeBytes doesn't work on OS X
Date Wed, 08 Feb 2012 14:52:00 GMT
Hi all,

I'm running into a problem with BinaryEncoder.writeBytes.  I'm trying to
wrap some documents in an Avro structure and then write them into an Avro
contain file.

My code looks roughly like this :

        File file = new File(...);
        ClassLoader cl = SimpleAvroExample.class.
getClassLoader();
        InputStream isSimpleWrapperSchema=
cl.getResourceAsStream("simple_wrapper.avpr");

        Schema.Parser parser = new Schema.Parser();
        Schema schemaSimpleWrapper = parser.parse(isSimpleWrapperSchema);
        GenericDatumWriter<GenericRecord> gdw = new
GenericDatumWriter<GenericRecord>(schemaSimpleWrapper);
        DataFileWriter writer = new DataFileWriter(gdw);

        File outputFile = new File(...);
        outputFile.createNewFile();

        writer.create(schemaSimpleWrapper, outputFile);


        for(File documentFile : file.listFiles())
        {

            GenericRecord r =  new GenericData.Record(schemaSimpleWrapper);
            FileInputStream fis = new FileInputStream(documentFile);
            FileChannel channel = fis.getChannel();

            r.put("filename", documentFile.getName());

             //Copying the whole file.
            ByteBuffer byteBuffer =
channel.map(FileChannel.MapMode.READ_ONLY, 0, documentFile.length());
            r.put("body", byteBuffer);




            writer.append(r);

            channel.close();
            fis.close();

        }

This code exceptions out :
      Exception in thread "main"
org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.UnsupportedOperationException
        at
org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:261)
        at org.iqt.cdl.SimpleAvroExample.main(SimpleAvroExample.java:107)
     Caused by: java.lang.UnsupportedOperationException
        at java.nio.ByteBuffer.arrayOffset(ByteBuffer.java:968)
       at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:61)
       at
org.apache.avro.generic.GenericDatumWriter.writeBytes(GenericDatumWriter.java:199)
       at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:76)
       at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
       at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
       at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
       at
org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:255)
       ... 1 more


The problem seems to be that the writeBytes function on line 61 does the
following :
     int start = bytes.arrayOffset() + pos.

Where byes is a java.nio.ByteBuffer.

According to the JavaDocs for ByteBuffer the arrayOffset fuctional is
optional.

As near as I can tell this OS X implementation of ByteBuffer does not
support his functionality.


-- 
Karthik Ramachandran
Mobile: 412-606-8981

Mime
View raw message