Mailing-List: contact dev-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@avro.apache.org
Message-ID: <556757A2.7030703@cloudera.com>
Date: Thu, 28 May 2015 11:00:02 -0700
From: Ryan Blue <blue@cloudera.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.7.0
MIME-Version: 1.0
To: dev@avro.apache.org
Subject: Re: Need review/merges for couple of pull requests
References: 
 <CAGX0URg-K=nzguJ7TRwuHAHT7TB+aVr-0_n0o24XPu7cdNuBzg@mail.gmail.com>
	<5512E56B.3010704@cloudera.com>
	<CAGX0URjzgRsT_7_GSqjKbZPUQX4jEv9d7=qBoBX3=DmqJhsPXg@mail.gmail.com>
	<CAGX0URgFjDQQnm8UWYJJWTMDJVb0gokb60UseNjszryNo==zdA@mail.gmail.com>
	<556644F3.2050509@cloudera.com>
 <CAGX0URgpnBt5rutbR=XS5-_5YpLVT5JoRbdi0hHc7fg30it+QA@mail.gmail.com>
In-Reply-To: 
 <CAGX0URgpnBt5rutbR=XS5-_5YpLVT5JoRbdi0hHc7fg30it+QA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

SG,

The data ends up looking like this:

{"id":1,"p":"parent data!","child":{"c":"child data!","parent":{"long":1}}}

I produced that with avro-tools 1.7.6 and tojson.

Here's the schema: {
   "type" : "record",
   "name" : "Parent",
   "fields" : [ {
     "name" : "id",
     "type" : "long"
   }, {
     "name" : "p",
     "type" : "string"
   }, {
     "name" : "child",
     "type" : {
       "type" : "record",
       "name" : "Child",
       "fields" : [ {
         "name" : "c",
         "type" : "string"
       }, {
         "name" : "parent",
         "type" : [ "null", "long", "Parent" ]
       } ],
       "logicalType" : "reference",
       "ref-field-name" : "parent"
     }
   } ],
   "logicalType" : "referenceable",
   "id-field-name" : "id"
}

rb

On 05/28/2015 09:50 AM, S G wrote:
> RB,
>
> Could you please attach the schema and the JSON serialized output from your
> test-code as well?
> My build environment is currently broken as I am grappling with some Java 8
> update issues.
>
> Thanks
> SG
>
>
> On Wed, May 27, 2015 at 3:28 PM, Ryan Blue <blue@cloudera.com> wrote:
>
>> SG,
>>
>> Now that logical types are in, I had some time to look at this issue.
>> Thanks for your patience on this.
>>
>> When I started looking at the use case, this began to look very much like
>> a logical type issue. (I know, I've been saying that a lot.) When you
>> write, you replace any referenced object with its id. When you read, you
>> replace those ids with the correct object. I went ahead and implemented
>> this using 2 logical types: Referenceable for an object with an id, and
>> Reference for a record with a field that references another object by id.
>>
>> There were some trade-offs to this approach. For example, I had to use a
>> logical type for the object containing the reference rather than for the
>> reference itself because I used a callback to set the object once it is
>> found. That happens because children with references to a parent are
>> deserialized completely first. The parent is the last object to be
>> assembled and passed to the logical type conversion.
>>
>> A bigger issue was that logical types are currently conservative and don't
>> overlap with reflect types or anything that sets java-class. That means
>> that this currently only works with generic types. But, I'd rather make
>> logical types work for reflect than add more custom code to support this.
>> Does that sound reasonable?
>>
>> I'm attaching a diff with the working test code so you can take a look at
>> the approach. Let me know what you are thinking.
>>
>> rb
>>
>> On 05/20/2015 12:05 PM, S G wrote:
>>
>>> I am requesting some help with AVRO-695.
>>> Here are some pieces from the last conversation.
>>>
>>>
>>> Doug Cutting
>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cutting>
>>> added
>>> a comment - 02/Oct/14 21:19
>>>
>>> Here's a modified version of the patch. It moves all significant changes
>>> to
>>> reflect. Since reflect is the only implementation that can generate an
>>> appropriate schema, changes should be confined to there.
>>>
>>> The tests need to be updated, as they still assume generic.
>>> <https://issues.apache.org/jira/browse/AVRO-695#>
>>> <
>>> https://issues.apache.org/jira/browse/AVRO-695?focusedCommentId=14286370&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14286370
>>>>
>>> Sachin Goyal
>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sachingoyal>
>>> added
>>> a comment - 21/Jan/15 21:56
>>>
>>> Here is a patch with the updated test-cases.
>>> I also confirm that all my changes are there in the patch.
>>>
>>>
>>> The patch was submitted in June 2014 and was very hot till October 2014.
>>> Since then, there has been no action on this even though I have sent many
>>> reminders in this group.
>>>
>>> I understand that everyone is very busy with their own stuff but I would
>>> really appreciate if someone could help a fellow engineer in getting his
>>> patch accepted.
>>> It would encourage more participation as well as help people wanting to
>>> use
>>> Avro for circular references.
>>>
>>> Regards
>>> SG
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Cloudera, Inc.
>>
>


-- 
Ryan Blue
Software Engineer
Cloudera, Inc.