incubator-crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Build Inconsistencies with Profiles
Date Tue, 22 Jan 2013 23:53:55 GMT
I think I have a fix, tracking it at:
https://issues.apache.org/jira/browse/CRUNCH-148


On Tue, Jan 22, 2013 at 3:17 PM, Josh Wills <jwills@cloudera.com> wrote:

> Yeah, I'm getting it too. I'll dive in and take a look.
>
> FYI, I usually build with the "-Dcrunch.platform=2" option.
>
>
> On Tue, Jan 22, 2013 at 3:01 PM, Whitacre,Micah <MICAH.WHITACRE@cerner.com
> > wrote:
>
>> For my code I'm maintaining a fork of Crunch simply because I target the
>> CDH4.1.x endstates.  When consuming the latest changes the tests are
>> however failing.
>>
>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.113 sec
>> <<< FAILURE!
>> testJoin(org.apache.crunch.lib.join.MultiAvroSchemaJoinIT)  Time elapsed:
>> 1.113 sec  <<< ERROR!
>> org.apache.crunch.CrunchRuntimeException: java.io.IOException: No files
>> found to materialize at:
>> /var/folders/0f/l_2w0gxd0p15k9410b18j8q40000gp/T/junit6697703972065781019/tmp-crunch.tmp.dir/crunch-1096140411/p1
>>         at
>> org.apache.crunch.materialize.MaterializableIterable.materialize(MaterializableIterable.java:70)
>>         at org.apache.crunch.impl.mr.MRPipeline.run(MRPipeline.java:158)
>>         at
>> org.apache.crunch.materialize.MaterializableIterable.iterator(MaterializableIterable.java:59)
>>         at com.google.common.collect.Lists.newArrayList(Lists.java:119)
>>         at
>> org.apache.crunch.lib.join.MultiAvroSchemaJoinIT.testJoin(MultiAvroSchemaJoinIT.java:116)
>>
>> The attachment are the only changes I've made to project and are just the
>> addition of a profile in the root pom.xml.  To execute I ran "mvn clean
>> install -P cdh4".  Reviewing the git log I don't see any major changes with
>> regard to how materialize or join were changed which seems to be related to
>> the issue.  I think it might be related to CRUNCH-127 as that was the only
>> major change for writing data.
>>
>> Trying to compare my profile against the two existing I have noticed
>> failures when building one of the profiles.  Specifically if I run "mvn
>> clean install" the build will run and complete successfully.  However if I
>> specify the "hadoop-2" profile it fails with a similar exception.
>>
>> Executing the following at the root of the git repository.
>> mvn clean install -P hadoop-2
>>
>> I get a failure during integration testing of crunch.
>>
>> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.52 sec
>> <<< FAILURE!
>> testMapsideJoin(org.apache.crunch.lib.join.MapsideJoinIT)  Time elapsed:
>> 2.434 sec  <<< ERROR!
>> org.apache.crunch.CrunchRuntimeException: java.io.IOException: No files
>> found to materialize at:
>> /tmp/junit1288475102834326447/tmp-crunch.tmp.dir/crunch-2036458765/p3
>>         at
>> org.apache.crunch.materialize.MaterializableIterable.materialize(MaterializableIterable.java:70)
>>         at org.apache.crunch.impl.mr.MRPipeline.run(MRPipeline.java:158)
>>         at
>> org.apache.crunch.lib.join.MapsideJoinIT.runMapsideJoin(MapsideJoinIT.java:139)
>>         at
>> org.apache.crunch.lib.join.MapsideJoinIT.testMapsideJoin(MapsideJoinIT.java:116)
>>
>> Details about my current setup for building:
>>
>> Git commit 75ba3546810fc27b4a6248559fefffd6c3eddb44
>>
>> $ mvn -v
>> Apache Maven 3.0.4 (r1232337; 2012-01-17 02:44:56-0600)
>> Maven home: /usr/local/Cellar/maven/3.0.4/libexec
>> Java version: 1.6.0_37, vendor: Apple Inc.
>> Java home:
>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>> Default locale: en_US, platform encoding: MacRoman
>> OS name: "mac os x", version: "10.7.5", arch: "x86_64", family: "mac"
>>
>> I accept responsibility about maintaining the CDH4 support but I'm
>> curious, am I building the project correctly?  Is this a local setup issue
>> and not something others encounter as well?
>>
>>
>> CONFIDENTIALITY NOTICE This message and any included attachments are from
>> Cerner Corporation and are intended only for the addressee. The information
>> contained in this message is confidential and may constitute inside or
>> non-public information under international, federal, or state securities
>> laws. Unauthorized forwarding, printing, copying, distribution, or use of
>> such information is strictly prohibited and may be unlawful. If you are not
>> the addressee, please promptly delete this message and notify the sender of
>> the delivery error by e-mail or you may call Cerner's corporate offices in
>> Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message