avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yibing Shi <y...@cloudera.com>
Subject Re: serialize pojo that contains a byte array
Date Tue, 22 Nov 2016 06:13:00 GMT
Hi Paul,

It looks like this problem is because Avro uses ByteBuffer for "bytes"
type. GenericData treats "byte[]" as an array, but ReflectData generates an
union type (["null",{"type":"bytes","java-class":"[B"}]).

Is it possible for you to wrap your byte[] in a ByteBuffer?

*Yibing Shi*
*Customer Operations Engineer*
<http://www.cloudera.com>

On Sat, Nov 19, 2016 at 1:24 AM, Paul Read <pdread101@gmail.com> wrote:

> If this is bad form I apologize up front, but I thought I should make some
> additional clarifications:
>
> 1. using GenericDatumWriter/Reader is not applicable for my application
> 2. I tried SpecificDatumWriter/Reader and got IndexRecord Exceptions
> 3. Changing it from byte[] to Byte[] will work but I need it to be as fast
> as possible to handle processing needs so converting it to Byte[] is
> something I would like to avoid.
>
> Thanks
>
>
> On Fri, Nov 18, 2016 at 7:29 AM, Paul Read <pdread101@gmail.com> wrote:
>
>> Sorry hit the send button accidently...to finish
>>
>> Unit Test
>>
>> public class  TestAvroHelper {
>>
>>       @Test
>>       public void testAvroHelper() {
>>           AvroHelper<TestAvroData> helper = new
>> AvroHelper<TestAvroHelper>(TestAvroHelper.class);
>>
>>             TestAvroData data = new TestAvroData();
>>             data.setData(new String("hello").toBytes());
>>
>>             byte adata = helper.toAvroBytes(data);
>>             TestAvroData data1 = helper.fromAvroBytes(adata);
>>
>>       }
>> }
>>
>> This fails with an Union Exception.
>>
>> What am I doing wrong?
>>
>> Thanks
>>
>>
>>
>> On Fri, Nov 18, 2016 at 7:25 AM, Paul Read <pdread101@gmail.com> wrote:
>>
>>>
>>> I have a AvroHelper that serializes/deserializes about 99% of some very
>>> complex classes, The one class component it cannot seem to handle is a
>>> field that contains a simple byte array. The helper is defined
>>>
>>> public class AvroHelper<T> {
>>>
>>>     private Class<T> type;
>>>     private Schema schema;
>>>     private DatumReader<T> reader;
>>>     private DatumWriter<T> writer;
>>>     private ByteArrayOutputStream bos;
>>>     private Encoder encoder;
>>>     private DecoderFactory decoderFactory;
>>>
>>>     public AvroHelper(Class<T> type) {
>>>
>>>            this.type = type;
>>>            this.schema = ReflectData.AllowNull.get().getSchema(type);
>>>            this.bos = new ByteArrayOutputStream();
>>>            this.decoderFactory = DencoderFactory.get();
>>>            this.encoder = EncoderFactory.get().binaryEncoder(bos, null);
>>>            this.reader = new ReflectDatumReader<T>(schema);
>>>            this.writer = new ReflectDatumWriter<T>(schema);
>>>     }
>>>
>>>     public byte[] toAvroBytes( T o) throws IOException {
>>>            bos.reset();
>>>            writer.write(o, encoder);
>>>            encoder.flush();
>>>            return bos.toByteArray();
>>>     }
>>>
>>>     public T fromAvroBytes(byte [] raw) throws IOException {
>>>               return reader.read(null, decoderFactory.binaryDecoder(raw,
>>> null));
>>>     }
>>> }
>>>
>>> The simple test POJO is
>>>
>>> public class TestAvroData {
>>>          private byte [] data;
>>>
>>>          public TestAvroData() {
>>>          }
>>>
>>>          public void setData(byte [] data) {
>>>                this.data = data;
>>>          }
>>>
>>>          public byte[] getData() {
>>>                  return this.data;
>>>          }
>>> }
>>>
>>> Unit Test
>>>
>>> public class  TestAvroHelper {
>>>
>>>       @Test
>>>       public void testAvroHelper() {
>>>           AvroHelper<TestAvroData> helper = new
>>> AvroHelper<TestAvroHelper>(TestAvroHelper.class);
>>>
>>>
>>>       }
>>>
>>
>>
>

Mime
View raw message