incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-24) Make test suite suitable for continuous integration
Date Mon, 23 Jul 2012 20:43:35 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420951#comment-13420951
] 

Josh Wills commented on CRUNCH-24:
----------------------------------

If I remove the Configuration argument, it works. The exception is mainly what you would expect--
the CrunchMapper cannot find the file that it tried to store in the DistributedCache:

6526 [Thread-26] INFO  org.apache.crunch.impl.mr.exec.CrunchJob  - Job status available at:
http://localhost:8080/
6564 [Thread-46] WARN  org.apache.hadoop.mapred.LocalJobRunner  - job_local_0002
org.apache.crunch.impl.mr.run.CrunchRuntimeException: Error reading right-side of map side
join: 
	at org.apache.crunch.lib.join.MapsideJoin$MapsideJoinDoFn.initialize(MapsideJoin.java:134)
	at org.apache.crunch.DoFn.setContext(DoFn.java:100)
	at org.apache.crunch.impl.mr.run.RTNode.initialize(RTNode.java:61)
	at org.apache.crunch.impl.mr.run.RTNode.initialize(RTNode.java:63)
	at org.apache.crunch.impl.mr.run.RTNode.initialize(RTNode.java:63)
	at org.apache.crunch.impl.mr.run.CrunchTaskContext.getNodes(CrunchTaskContext.java:56)
	at org.apache.crunch.impl.mr.run.CrunchMapper.setup(CrunchMapper.java:40)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
Caused by: java.io.IOException: No files found to materialize at: file:/tmp/hadoop-josh/mapred/local/archive/313161001601735375_-1266581102_898898625/file/var/folders/ga/gauyLIkgEiuMA30hW9H3YU+++TI/-Tmp-/junit7903276863692299885/junit3527640565817378898/crunch-1754643543/p1
	at org.apache.crunch.io.CompositePathIterable.create(CompositePathIterable.java:48)
	at org.apache.crunch.io.seq.SeqFileTableSource.read(SeqFileTableSource.java:48)
	at org.apache.crunch.io.impl.ReadableSourcePathTargetImpl.read(ReadableSourcePathTargetImpl.java:35)
	at org.apache.crunch.lib.join.MapsideJoin$MapsideJoinDoFn.initialize(MapsideJoin.java:132)
	... 10 more
                
> Make test suite suitable for continuous integration
> ---------------------------------------------------
>
>                 Key: CRUNCH-24
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-24
>             Project: Crunch
>          Issue Type: Task
>    Affects Versions: 0.3.0
>            Reporter: Matthias Friedrich
>            Assignee: Matthias Friedrich
>             Fix For: 0.3.0
>
>         Attachments: 0001-CRUNCH-24-Clean-up-test-suite-output.patch, 0001-CRUNCH-24-make-testsuite-sutable-for-CI.patch,
0001-CRUNCH-24-make-testsuite-sutable-for-CI.patch
>
>
> Right now the integration test suite leaves about 80 files behind on /tmp making it unsuitable
for a shared continuous integration environment. Examples for these files are test case output
("output*") and Crunch's own temporary files ("crunch*", see CRUNCH-21).
> We have to delete these files or make sure they aren't created on /tmp in the first place.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message