Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-dev@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of tom@cloudera.com designates
 209.85.160.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAH29n6NO7hav5F_CDFYaZUCd49aMC482ksaGOX=4M=cuOE5ZeA@mail.gmail.com>
References: 
 <CAH29n6NEnrXo-d6LZVbBPkHar0bQ4x5toiQ9P7ji6NzmOV533w@mail.gmail.com>
 <CAA5C_pujXD4bQ-zzRc2hK1A+d6H3wHHawQu0=nwgwQCtA3mo3A@mail.gmail.com>
 <CAH29n6NO7hav5F_CDFYaZUCd49aMC482ksaGOX=4M=cuOE5ZeA@mail.gmail.com>
From: Tom White <tom@cloudera.com>
Date: Tue, 12 Jun 2012 15:47:44 -0500
Message-ID: 
 <CAF-WD4TJ6fAHYYUFqts0LeSRbvY8f7oE_h+Bs8muH+90SNh86g@mail.gmail.com>
Subject: Re: avro 1.7.0: should we upgrade crunch?
To: Josh Wills <jwills@cloudera.com>
Cc: crunch-dev@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Funny - I was just writing an email about potential dragons...

Before upgrading someone should check that Avro 1.7.0 works with
released versions of Hadoop. In the past there have been problems with
Avro and dependencies like Jackson conflicting with versions that
Hadoop uses. In particular, the MR classpath can be controlled via the
configuration property mapreduce.user.classpath.first and the env
property HADOOP_USER_CLASSPATH_FIRST. By setting these to true MR will
use the newer Avro libraries, however there is a risk that Hadoop will
not work with the newer versions.

Cheers,
Tom

On Tue, Jun 12, 2012 at 3:36 PM, Josh Wills <jwills@cloudera.com> wrote:
> On Tue, Jun 12, 2012 at 1:28 PM, Gabriel Reid <gabriel.reid@gmail.com> wrote:
>> On Tue, Jun 12, 2012 at 10:14 PM, Josh Wills <jwills@cloudera.com> wrote:
>>>
>>> I saw Chris' tweet that Avro 1.7.0 was released-- thoughts on
>>> upgrading? Does it buy us any good stuff we want?
>>
>> The fix of ReflectDatumReader not working correctly with Specific
>> Records (https://issues.apache.org/jira/browse/AVRO-1046) would allow
>> us to remove a fair bit of Avro code that works around that bug -- if
>> we do do the upgrade, I'd certainly volunteer to weed out those
>> workarounds.
>>
>> On the other hand, I just did a quick scan of the release notes for
>> 1.7.0 (https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310911&version=12318848)
>> and I didn't notice anything that would really be a big reason to
>> upgrade right away.
>>
>> Are there any reasons *not* to upgrade (other than risks of something
>> else being broken)? Maybe the cleanup of the Avro code that we can do
>> is reason enough to do the upgrade.
>
> +tom explicitly
>
> I remember having a bunch of frustrations with 1.6.0 and having to
> stay on 1.5.4 for longer than I wanted because of some critical bugs
> that didn't get fixed until 1.6.2, but I also think the move from
> 1.5.4 to 1.6.0 involved a much larger rewrite than what I see from the
> release notes for 1.7.0. Tom White is traveling across the US right
> now, but I'm wondering if he has a feel for whether 1.7.0 is likely to
> contain any dragons. :)
>
>>
>> - Gabriel
>
>
>
> --
> Director of Data Science
> Cloudera
> Twitter: @josh_wills