incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject Re: pipeline.done with pcollection.materialize in tests, why ?
Date Thu, 26 Jul 2012 13:34:35 GMT
Pipeline.done does some cleanup, like deleting the temp directories that
contain any intermediate output Crunch generated, including the
materialized data. With the changes in CRUNCH-24 to move all of the
temporary directories to use JUnit's TemporaryFolder, I don't think the
done() calls are necessary, but they aren't doing any harm.

On Thu, Jul 26, 2012 at 4:27 AM, Rahul <> wrote:

> Hi,
> I have seen tests where /Pipeline.done()/ was being called in the /@After/
> method  of a junit or at the end of test after all assertions have been
> done.
> All of these tests were using the /materialize() /API to get the
> collection data.
> Now, why is /pipeline.done()/  is being called in such cases ? Does it
> close some open ends, if so then which ones ?
> regards
> Rahul

Director of Data Science
Cloudera <>
Twitter: @josh_wills <>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message