avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <sc...@richrelevance.com>
Subject Re: reading back, "type" : "bytes" from java via ByteBuffer
Date Mon, 21 Mar 2011 16:53:41 GMT
Yes, it should conform to the standard ByteBuffer API:
http://download.oracle.com/javase/1.5.0/docs/api/java/nio/ByteBuffer.html
http://download.oracle.com/javase/1.5.0/docs/api/java/nio/Buffer.html

The limit is set to the end of the valid data in the array.  Since Avro
returns ByteBuffers with position() and arrayOffset() always set to zero,
limit() and remaining() return the same value -- which is equal to the
length of valid data from the start of the backing array.


On 3/18/11 9:02 PM, "sitkack@gmail.com" <sitkack@gmail.com> wrote:

>
>I can confirm that the code below functions properly. I was seeing the
>BufferUnderflowException because of a spurious bit of debug code in
>another object.
>
>On Mar 18, 2011, at 7:57 PM, sitkack@gmail.com wrote:
>
>> Using the following I get an java.nio.BufferUnderflowException
>> 
>> // where pdf is a ByteBuffer from my Avro stream
>> int size = pdf.remaining();
>> byte[] buf = new byte[size];
>> pdf.get(buf,0,size);
>> 
>> The pathology I am currently seeing is when I write a file out (from
>>data contained in the Avro) it has trailing data from the previous
>>larger file.
>> 
>> data = [[smaller file] extra data from previous file ]
>> 
>> contained in the ByteBuffer
>> 
>> 
>> On Mar 18, 2011, at 7:25 PM, David Rosenstrauch wrote:
>> 
>>> I think - and someone please correct me if I'm wrong - the offset is
>>>always zero, and the length is byteBuffer.remaining().
>>> 
>>> So you would make a call something like:
>>> 
>>> byteBuffer.get(byteArray, 0, byteBuffer.remaining())
>>> 
>>> Then byteArray would contain the buffer's contents.
>>> 
>>> HTH,
>>> 
>>> DR
>>> 
>>> On 03/18/2011 08:22 PM, sean jensen-grey wrote:
>>>> I have a large sequence of pdfs stored in an avro file as part of a
>>>>larger structure.
>>>> 
>>>> I have found a bug in my code where I was calling
>>>> 
>>>>    byteBuffer.array() to get back the byte[], this is incorrect as
>>>>this is entire backing store and NOT the contents of the element
>>>>stored in Avro.
>>>> 
>>>> How/where do I get the offset and the length of the ByteBuffer
>>>>returned from Avro?
>>>> 
>>>> The convenience classes were generated via the maven plugin so my
>>>>Record signature is
>>>> 
>>>>     MyRecord extends org.apache.avro.specific.SpecificRecordBase
>>>>implements org.apache.avro.specific.SpecificRecord
>>>> 
>>>> The avro schema entry is
>>>> 
>>>> {
>>>>     "name" : "pdfs",
>>>>     "type" :  {  "type" : "array", "items": "bytes" }
>>>> }
>>> 
>> 
>


Mime
View raw message