avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Read <pdread...@gmail.com>
Subject Re: serialize pojo that contains a byte array
Date Tue, 22 Nov 2016 09:35:52 GMT
Hi

I tried ByteBuffer and it did not like that either. I finally settled on
Byte[] but I'm wondering why its not smart enough to handle that construct.
The Avro spec seems to imply that it should but I'm not sure if I'm reading
it correctly. Bug maybe?

Thanks

Paul

On Tue, Nov 22, 2016 at 1:13 AM, Yibing Shi <yshi@cloudera.com> wrote:

> Hi Paul,
>
> It looks like this problem is because Avro uses ByteBuffer for "bytes"
> type. GenericData treats "byte[]" as an array, but ReflectData generates an
> union type (["null",{"type":"bytes","java-class":"[B"}]).
>
> Is it possible for you to wrap your byte[] in a ByteBuffer?
>
> *Yibing Shi*
> *Customer Operations Engineer*
> <http://www.cloudera.com>
>
> On Sat, Nov 19, 2016 at 1:24 AM, Paul Read <pdread101@gmail.com> wrote:
>
>> If this is bad form I apologize up front, but I thought I should make
>> some additional clarifications:
>>
>> 1. using GenericDatumWriter/Reader is not applicable for my application
>> 2. I tried SpecificDatumWriter/Reader and got IndexRecord Exceptions
>> 3. Changing it from byte[] to Byte[] will work but I need it to be as
>> fast as possible to handle processing needs so converting it to Byte[] is
>> something I would like to avoid.
>>
>> Thanks
>>
>>
>> On Fri, Nov 18, 2016 at 7:29 AM, Paul Read <pdread101@gmail.com> wrote:
>>
>>> Sorry hit the send button accidently...to finish
>>>
>>> Unit Test
>>>
>>> public class  TestAvroHelper {
>>>
>>>       @Test
>>>       public void testAvroHelper() {
>>>           AvroHelper<TestAvroData> helper = new
>>> AvroHelper<TestAvroHelper>(TestAvroHelper.class);
>>>
>>>             TestAvroData data = new TestAvroData();
>>>             data.setData(new String("hello").toBytes());
>>>
>>>             byte adata = helper.toAvroBytes(data);
>>>             TestAvroData data1 = helper.fromAvroBytes(adata);
>>>
>>>       }
>>> }
>>>
>>> This fails with an Union Exception.
>>>
>>> What am I doing wrong?
>>>
>>> Thanks
>>>
>>>
>>>
>>> On Fri, Nov 18, 2016 at 7:25 AM, Paul Read <pdread101@gmail.com> wrote:
>>>
>>>>
>>>> I have a AvroHelper that serializes/deserializes about 99% of some very
>>>> complex classes, The one class component it cannot seem to handle is a
>>>> field that contains a simple byte array. The helper is defined
>>>>
>>>> public class AvroHelper<T> {
>>>>
>>>>     private Class<T> type;
>>>>     private Schema schema;
>>>>     private DatumReader<T> reader;
>>>>     private DatumWriter<T> writer;
>>>>     private ByteArrayOutputStream bos;
>>>>     private Encoder encoder;
>>>>     private DecoderFactory decoderFactory;
>>>>
>>>>     public AvroHelper(Class<T> type) {
>>>>
>>>>            this.type = type;
>>>>            this.schema = ReflectData.AllowNull.get().getSchema(type);
>>>>            this.bos = new ByteArrayOutputStream();
>>>>            this.decoderFactory = DencoderFactory.get();
>>>>            this.encoder = EncoderFactory.get().binaryEncoder(bos,
>>>> null);
>>>>            this.reader = new ReflectDatumReader<T>(schema);
>>>>            this.writer = new ReflectDatumWriter<T>(schema);
>>>>     }
>>>>
>>>>     public byte[] toAvroBytes( T o) throws IOException {
>>>>            bos.reset();
>>>>            writer.write(o, encoder);
>>>>            encoder.flush();
>>>>            return bos.toByteArray();
>>>>     }
>>>>
>>>>     public T fromAvroBytes(byte [] raw) throws IOException {
>>>>               return reader.read(null, decoderFactory.binaryDecoder(raw,
>>>> null));
>>>>     }
>>>> }
>>>>
>>>> The simple test POJO is
>>>>
>>>> public class TestAvroData {
>>>>          private byte [] data;
>>>>
>>>>          public TestAvroData() {
>>>>          }
>>>>
>>>>          public void setData(byte [] data) {
>>>>                this.data = data;
>>>>          }
>>>>
>>>>          public byte[] getData() {
>>>>                  return this.data;
>>>>          }
>>>> }
>>>>
>>>> Unit Test
>>>>
>>>> public class  TestAvroHelper {
>>>>
>>>>       @Test
>>>>       public void testAvroHelper() {
>>>>           AvroHelper<TestAvroData> helper = new
>>>> AvroHelper<TestAvroHelper>(TestAvroHelper.class);
>>>>
>>>>
>>>>       }
>>>>
>>>
>>>
>>
>

Mime
View raw message