Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
Received-SPF: pass (athena.apache.org: domain of grep.alex@gmail.com
 designates 209.85.216.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAK0mCrRpYbKZ5ymMTBeLeM6pqnim7qLcWuQDbKOrC0FdrMnZ1g@mail.gmail.com>
References: 
 <CAK0mCrT6Z-_uD=69Mnddf51sFf_2LysrH7rCtVLHoXCx44Yx-A@mail.gmail.com>
	<CA9D7E60.51D29%scott@richrelevance.com>
	<CAK0mCrRpYbKZ5ymMTBeLeM6pqnim7qLcWuQDbKOrC0FdrMnZ1g@mail.gmail.com>
Date: Tue, 20 Sep 2011 06:44:05 -0400
Message-ID: 
 <CAK0mCrQ5kZ=sSz3a8aWy77Nf7O+51cAMmyd4z=YgCR8YZB0Q6w@mail.gmail.com>
Subject: Re: Avro versioning and SpecificDatum's
From: Alex Holmes <grep.alex@gmail.com>
To: user@avro.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Created the following ticket:

https://issues.apache.org/jira/browse/AVRO-891

Thanks,
Alex

On Tue, Sep 20, 2011 at 6:26 AM, Alex Holmes <grep.alex@gmail.com> wrote:
> Thanks, I'll add a bug.
>
> As a FYI, even without the alias (retaining the original field name),
> just removing the "id" field yields the exception.
>
> On Tue, Sep 20, 2011 at 2:22 AM, Scott Carey <scottcarey@apache.org> wrot=
e:
>> That looks like a bug. =A0What happens if there is no aliasing/renaming
>> involved? =A0Aliasing is a newer feature than field addition, removal, a=
nd
>> promotion.
>>
>> This should be easy to reproduce, can you file a JIRA ticket? =A0We shou=
ld
>> discuss this further there.
>>
>> Thanks!
>>
>>
>> On 9/19/11 6:14 PM, "Alex Holmes" <grep.alex@gmail.com> wrote:
>>
>>>OK, I was able to reproduce the exception.
>>>
>>>v1:
>>>{"name": "Record", "type": "record",
>>> =A0"fields": [
>>> =A0 =A0{"name": "name", "type": "string"},
>>> =A0 =A0{"name": "id", "type": "int"}
>>> =A0]
>>>}
>>>
>>>v2:
>>>{"name": "Record", "type": "record",
>>> =A0"fields": [
>>> =A0 =A0{"name": "name_rename", "type": "string", "aliases": ["name"]}
>>> =A0]
>>>}
>>>
>>>Step 1. =A0Write Avro file using v1 generated class
>>>Step 2. =A0Read Avro file using v2 generated class
>>>
>>>Exception in thread "main" org.apache.avro.AvroRuntimeException: Bad ind=
ex
>>> =A0 =A0 =A0 at Record.put(Unknown Source)
>>> =A0 =A0 =A0 at org.apache.avro.generic.GenericData.setField(GenericData=
.java:463)
>>> =A0 =A0 =A0 at
>>>org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader=
.j
>>>ava:166)
>>> =A0 =A0 =A0 at
>>>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:=
13
>>>8)
>>> =A0 =A0 =A0 at
>>>org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:=
12
>>>9)
>>> =A0 =A0 =A0 at org.apache.avro.file.DataFileStream.next(DataFileStream.=
java:233)
>>> =A0 =A0 =A0 at org.apache.avro.file.DataFileStream.next(DataFileStream.=
java:220)
>>> =A0 =A0 =A0 at Read.readFromAvro(Unknown Source)
>>> =A0 =A0 =A0 at Read.main(Unknown Source)
>>>
>>>The code to write/read the avro file didn't change from below.
>>>
>>>On Mon, Sep 19, 2011 at 9:08 PM, Alex Holmes <grep.alex@gmail.com> wrote=
:
>>>> I'm trying to put together a simple test case to reproduce the
>>>> exception. =A0While I was creating the test case, I hit this behavior
>>>> which doesn't seem right, but maybe it's my misunderstanding on how
>>>> forward/backward compatibility should work:
>>>>
>>>> Schema v1:
>>>>
>>>> {"name": "Record", "type": "record",
>>>> =A0"fields": [
>>>> =A0 =A0{"name": "name", "type": "string"},
>>>> =A0 =A0{"name": "id", "type": "int"}
>>>> =A0]
>>>> }
>>>>
>>>> Schema v2:
>>>>
>>>> {"name": "Record", "type": "record",
>>>> =A0"fields": [
>>>> =A0 =A0{"name": "name_rename", "type": "string", "aliases": ["name"]},
>>>> =A0 =A0{"name": "new_field", "type": "int", "default":"0"}
>>>> =A0]
>>>> }
>>>>
>>>> In the 2nd version I:
>>>>
>>>> - removed field "id"
>>>> - renamed field "name" to "name_rename"
>>>> - added field "new_field"
>>>>
>>>> I write the v1 data file:
>>>>
>>>> =A0public static Record createRecord(String name, int id) {
>>>> =A0 =A0Record record =3D new Record();
>>>> =A0 =A0record.name =3D name;
>>>> =A0 =A0record.id =3D id;
>>>> =A0 =A0return record;
>>>> =A0}
>>>>
>>>> =A0public static void writeToAvro(OutputStream outputStream)
>>>> =A0 =A0 =A0throws IOException {
>>>> =A0 =A0DataFileWriter<Record> writer =3D
>>>> =A0 =A0 =A0 =A0new DataFileWriter<Record>(new SpecificDatumWriter<Reco=
rd>());
>>>> =A0 =A0writer.create(Record.SCHEMA$, outputStream);
>>>>
>>>> =A0 =A0writer.append(createRecord("r1", 1));
>>>> =A0 =A0writer.append(createRecord("r2", 2));
>>>>
>>>> =A0 =A0writer.close();
>>>> =A0 =A0outputStream.close();
>>>> =A0}
>>>>
>>>> I wrote a version-agnostic Read class:
>>>>
>>>> =A0public static void readFromAvro(InputStream is) throws IOException =
{
>>>> =A0 =A0DataFileStream<Record> reader =3D new DataFileStream<Record>(
>>>> =A0 =A0 =A0 =A0 =A0 =A0is, new SpecificDatumReader<Record>());
>>>> =A0 =A0for (Record a : reader) {
>>>> =A0 =A0 =A0System.out.println(ToStringBuilder.reflectionToString(a));
>>>> =A0 =A0}
>>>> =A0 =A0IOUtils.cleanup(null, is);
>>>> =A0 =A0IOUtils.cleanup(null, reader);
>>>> =A0}
>>>>
>>>> Running the Read code against the v1 data file, and including the v1
>>>> code-generated classes in the classpath produced:
>>>>
>>>> Record@6a8c436b[name=3Dr1,id=3D1]
>>>> Record@6baa9f99[name=3Dr2,id=3D2]
>>>>
>>>> If I run the same code, but use just the v2 generated classes in the
>>>> classpath I get:
>>>>
>>>> Record@39dd3812[name_rename=3Dr1,new_field=3D1]
>>>> Record@27b15692[name_rename=3Dr2,new_field=3D2]
>>>>
>>>> The name_rename field seems to be good, but why would "new_field"
>>>> inherit the values of the deleted field "id"?
>>>>
>>>> Cheers,
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Sep 19, 2011 at 12:35 PM, Doug Cutting <cutting@apache.org>
>>>>wrote:
>>>>> On 09/19/2011 05:12 AM, Alex Holmes wrote:
>>>>>> I then modified my original schema by adding, deleting and renaming
>>>>>> some fields, creating version 2 of the schema. =A0After re-creating =
the
>>>>>> Java classes I attempted to read the version 1 file using the
>>>>>> DataFileStream (with a SpecificDatumReader), and this is throwing an
>>>>>> exception.
>>>>>
>>>>> This should work. =A0Can you provide more detail? =A0What is the exce=
ption?
>>>>> =A0A reproducible test case would be great to have.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Doug
>>>>>
>>>>
>>
>>
>>
>