avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramana Suvarapu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1474) C++ resolvind decoder doesn't work when reader schema has more fields than writer schema
Date Thu, 24 Apr 2014 04:01:16 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979252#comment-13979252

Ramana Suvarapu commented on AVRO-1474:

Patch is not working in the following scenario.

Hi Thiru,

Attached are writer and reader schema. In writer schema I added new field “optionalNestedrecord”
to RootRecord which is optional and default value is null.  This field doesn’t exist in
Reader schema.

                                                "name": "optionalNestedrecord",
                                                "type": ["null", "Nested"],
                                                "default": null


When I populate this field in writers RootRecord and when I tried to deserialize using reader
schema, It’s throwing following exception.

> at memory location 0x00b5d3b0..

This exception is thrown in the following section of void skip(Decoder& d) of Symbol.hh

case Symbol::sSymbolic:
                    ProductionPtr pp(
                        t.extra<boost::weak_ptr<Production> >());

Steps to reproduce:
1.	Use the attached files to regenerate bigrecord types for both reader and writer
2.	In writer RootRecord, populate  optionalNestedrecord  and encode it
3.	Using ResolvingDecoder and using reader schema, try to deserialize encoded writer’s record
and you will see this exception.

I attached theses schema to AVRO-1474 jira. 
Please take a look into this. 


On Wed, Apr 23, 2014 at 7:41 PM, Srinidhi S <sri0406@gmail.com> wrote:
Hi Thiru

I found a bug with latest patch.  Here is the scenario.

writer record schema has optional field which is defined as union [null, another record] and
here "another record" is defined as Symbolic to schema instead of actual definition of full
record schema.

If reader schema doesn't have this field and writer populates this field and try to decode
writer's record using reader schema it's throwing boost::bad_ptr exception. The exception
is thrown from symbol class where it's trying to extract extra info from Production for Symbollic.

I have a test case to reproduce this issue and I will send it in my next email.


> C++ resolvind decoder doesn't work when reader schema has more fields than writer schema
> ----------------------------------------------------------------------------------------
>                 Key: AVRO-1474
>                 URL: https://issues.apache.org/jira/browse/AVRO-1474
>             Project: Avro
>          Issue Type: Bug
>          Components: c++
>    Affects Versions: 1.7.6
>            Reporter: Ramana Suvarapu
>            Assignee: Thiruvalluvan M. G.
>         Attachments: AVRO-1474.patch, bigrecord, bigrecord_r, reader, writer
> When reader schema has more number of fields than writer schema, C++ implementation of
resolving decoder is throwing exception "throwing exception "Don't know how to handle excess
fields for reader.” with out checking whether fields are optional or fields have default
> Attached are reader and writer schemas. Record in reader schema has 2 additional fields
than writer schema. One field is required field but it has default value and another one is
optional field (union of null and string). Since one has default value and another is optional
both reader and writer schemas are supposed to be compatible. 
> {"name": "defaultField", "type": "string", "default": "DEFAULT", "declared":"true"},
> {"name": "optionalField", "type": ["string", "null"],"declared":"true"},
> main()
> {
>   avro::ValidSchema readerSchema = load("reader.json");
>   avro::ValidSchema writerSchema = load("writer.json");
>   avro::DecoderPtr d = avro::resolvingDecoder(writerSchema, readerSchema,avro::binaryDecoder());
> }
> But when I tried to create resolving decoder, I am getting "Don't know how to handle
excess fields for reader.” But Java implementation works.  
> Also field ordering is not working. 
> The same issue is reported in AVRO-1360.

This message was sent by Atlassian JIRA

View raw message