crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Whitacre <mkwhita...@gmail.com>
Subject Re: Viewing intermediate states for debugging
Date Thu, 24 Jan 2013 20:24:16 GMT
I am creating an entirely new profile simply to keep my changes
separate from what is in apache/master.

Thanks for the hint about the "naive" approach.  Previously I had the following:

            <hadoop.version>2.0.0-cdh4.1.1</hadoop.version>
            <hadoop.client.version>2.0.0-mr1-cdh4.1.1</hadoop.client.version>
            <hbase.version>0.92.1-cdh4.1.1</hbase.version>

If I follow what you did and change it to:

            <hadoop.version>2.0.0-cdh4.1.1</hadoop.version>
            <hadoop.client.version>2.0.0-cdh4.1.1</hadoop.client.version>
            <hbase.version>0.92.1-cdh4.1.1</hbase.version>

The build gets farther.  I now have a different failure in
crunch-hbase I'll start working on.

Thanks for your help.
Micah


On Thu, Jan 24, 2013 at 12:23 PM, Josh Wills <jwills@cloudera.com> wrote:
> Micah,
>
> I did the naive thing and just swapped in 2.0.0-cdh4.1.2 for 2.0.0-alpha in
> the crunch.platform=2 profile in the top level POM and then added in the
> Cloudera repositories. That works for me-- does it work for you? It sounds
> to me like you're creating an entirely new profile.
>
> J
>
>
> On Thu, Jan 24, 2013 at 7:58 AM, Micah Whitacre <mkwhitacre@gmail.com>
> wrote:
>>
>> running dependency:tree on both projects shows that the version of
>> Avro is 1.7.0 for running under both profiles.  I wish it was that
>> easy.  :)
>>
>> On Thu, Jan 24, 2013 at 9:53 AM, Josh Wills <jwills@cloudera.com> wrote:
>> >
>> >
>> >
>> > On Thu, Jan 24, 2013 at 6:40 AM, Micah Whitacre <mkwhitacre@gmail.com>
>> > wrote:
>> >>
>> >> Taking a step back and comparing what is being generated for a normal
>> >> successful test run of "-Dcrunch.platform=2" I do see a p1 and p2
>> >> directory being created, with the expected materialized output being
>> >> in the p1 directory.  So I'm still curious about tracking all of the
>> >> intermediate state but it doesn't look like it is an issue with regard
>> >> to creating the output in the wrong directory.
>> >
>> >
>> > That's a relief. :)
>> >
>> > I think the issue with temp outputs has to do with our use of the
>> > TemporaryPath libraries for creating, well, temporary paths. We do this
>> > so
>> > we play nicely with CI frameworks, but you might need to disable it for
>> > investigating intermediate outputs.
>> >
>> > Re: the specific error you're seeing, that looks interesting. I wonder
>> > if
>> > it's an Avro version change or some such thing. Will see if I can
>> > replicate
>> > it.
>> >
>> >
>> > --
>> > Director of Data Science
>> > Cloudera
>> > Twitter: @josh_wills
>
>
>
>
> --
> Director of Data Science
> Cloudera
> Twitter: @josh_wills

Mime
View raw message