incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: avro 1.7.0: should we upgrade crunch?
Date Tue, 12 Jun 2012 22:57:21 GMT
I can put it through the paces w/cdh4, hadoop-1.0.3, and
hadoop-2.0.0-alpha, verify that tests pass, and then integrate the
pull req.

J

On Tue, Jun 12, 2012 at 3:55 PM, Christian Tzolov
<christian.tzolov@gmail.com> wrote:
> I am afraid I have pulled the Avro upgrade request far too soon. I should
> have checked my mailbox first.
>
> We have been using avro-1.7.0-rc0 for a week now to resolve AVRO-1046 issue.
> No problems were observed so far but we have not tested it with any other
> Hadoop version but chd3.
>
> Cheers,
> Chris
>
> On Tue, Jun 12, 2012 at 10:47 PM, Tom White <tom@cloudera.com> wrote:
>>
>> Funny - I was just writing an email about potential dragons...
>>
>> Before upgrading someone should check that Avro 1.7.0 works with
>> released versions of Hadoop. In the past there have been problems with
>> Avro and dependencies like Jackson conflicting with versions that
>> Hadoop uses. In particular, the MR classpath can be controlled via the
>> configuration property mapreduce.user.classpath.first and the env
>> property HADOOP_USER_CLASSPATH_FIRST. By setting these to true MR will
>> use the newer Avro libraries, however there is a risk that Hadoop will
>> not work with the newer versions.
>>
>> Cheers,
>> Tom
>>
>> On Tue, Jun 12, 2012 at 3:36 PM, Josh Wills <jwills@cloudera.com> wrote:
>> > On Tue, Jun 12, 2012 at 1:28 PM, Gabriel Reid <gabriel.reid@gmail.com>
>> > wrote:
>> >> On Tue, Jun 12, 2012 at 10:14 PM, Josh Wills <jwills@cloudera.com>
>> >> wrote:
>> >>>
>> >>> I saw Chris' tweet that Avro 1.7.0 was released-- thoughts on
>> >>> upgrading? Does it buy us any good stuff we want?
>> >>
>> >> The fix of ReflectDatumReader not working correctly with Specific
>> >> Records (https://issues.apache.org/jira/browse/AVRO-1046) would allow
>> >> us to remove a fair bit of Avro code that works around that bug -- if
>> >> we do do the upgrade, I'd certainly volunteer to weed out those
>> >> workarounds.
>> >>
>> >> On the other hand, I just did a quick scan of the release notes for
>> >> 1.7.0
>> >> (https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310911&version=12318848)
>> >> and I didn't notice anything that would really be a big reason to
>> >> upgrade right away.
>> >>
>> >> Are there any reasons *not* to upgrade (other than risks of something
>> >> else being broken)? Maybe the cleanup of the Avro code that we can do
>> >> is reason enough to do the upgrade.
>> >
>> > +tom explicitly
>> >
>> > I remember having a bunch of frustrations with 1.6.0 and having to
>> > stay on 1.5.4 for longer than I wanted because of some critical bugs
>> > that didn't get fixed until 1.6.2, but I also think the move from
>> > 1.5.4 to 1.6.0 involved a much larger rewrite than what I see from the
>> > release notes for 1.7.0. Tom White is traveling across the US right
>> > now, but I'm wondering if he has a feel for whether 1.7.0 is likely to
>> > contain any dragons. :)
>> >
>> >>
>> >> - Gabriel
>> >
>> >
>> >
>> > --
>> > Director of Data Science
>> > Cloudera
>> > Twitter: @josh_wills
>
>



-- 
Director of Data Science
Cloudera
Twitter: @josh_wills

Mime
View raw message