incubator-crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject Re: Viewing intermediate states for debugging
Date Thu, 24 Jan 2013 15:53:02 GMT
On Thu, Jan 24, 2013 at 6:40 AM, Micah Whitacre <>wrote:

> Taking a step back and comparing what is being generated for a normal
> successful test run of "-Dcrunch.platform=2" I do see a p1 and p2
> directory being created, with the expected materialized output being
> in the p1 directory.  So I'm still curious about tracking all of the
> intermediate state but it doesn't look like it is an issue with regard
> to creating the output in the wrong directory.

That's a relief. :)

I think the issue with temp outputs has to do with our use of the
TemporaryPath libraries for creating, well, temporary paths. We do this so
we play nicely with CI frameworks, but you might need to disable it for
investigating intermediate outputs.

Re: the specific error you're seeing, that looks interesting. I wonder if
it's an Avro version change or some such thing. Will see if I can replicate

Director of Data Science
Cloudera <>
Twitter: @josh_wills <>

View raw message