incubator-crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Viewing intermediate states for debugging
Date Thu, 24 Jan 2013 18:23:44 GMT
Micah,

I did the naive thing and just swapped in 2.0.0-cdh4.1.2 for 2.0.0-alpha in
the crunch.platform=2 profile in the top level POM and then added in the
Cloudera repositories. That works for me-- does it work for you? It sounds
to me like you're creating an entirely new profile.

J


On Thu, Jan 24, 2013 at 7:58 AM, Micah Whitacre <mkwhitacre@gmail.com>wrote:

> running dependency:tree on both projects shows that the version of
> Avro is 1.7.0 for running under both profiles.  I wish it was that
> easy.  :)
>
> On Thu, Jan 24, 2013 at 9:53 AM, Josh Wills <jwills@cloudera.com> wrote:
> >
> >
> >
> > On Thu, Jan 24, 2013 at 6:40 AM, Micah Whitacre <mkwhitacre@gmail.com>
> > wrote:
> >>
> >> Taking a step back and comparing what is being generated for a normal
> >> successful test run of "-Dcrunch.platform=2" I do see a p1 and p2
> >> directory being created, with the expected materialized output being
> >> in the p1 directory.  So I'm still curious about tracking all of the
> >> intermediate state but it doesn't look like it is an issue with regard
> >> to creating the output in the wrong directory.
> >
> >
> > That's a relief. :)
> >
> > I think the issue with temp outputs has to do with our use of the
> > TemporaryPath libraries for creating, well, temporary paths. We do this
> so
> > we play nicely with CI frameworks, but you might need to disable it for
> > investigating intermediate outputs.
> >
> > Re: the specific error you're seeing, that looks interesting. I wonder if
> > it's an Avro version change or some such thing. Will see if I can
> replicate
> > it.
> >
> >
> > --
> > Director of Data Science
> > Cloudera
> > Twitter: @josh_wills
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message