crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-400) Materialized jobs should have stage in PipelineResult
Date Tue, 27 May 2014 21:50:02 GMT


Josh Wills commented on CRUNCH-400:

Okay, I understand now. Calling:


is going to immediately materialize the results. And there's no way to get the StageResults
associated with doing that. The client-side fix would be to alter the code to run like this:

Iterable<String> preMaterialized = dataToBeMaterialized.materialize();
PipelineResult res =;
Set<String> materializedData = Sets.newHashSet(preMaterialized);

I could also make the MaterializableIterable that gets returned by the materialize() call
aware of its StageResults (I think).

> Materialized jobs should have stage in PipelineResult
> -----------------------------------------------------
>                 Key: CRUNCH-400
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.9.0, 0.8.2
>            Reporter: Micah Whitacre
> Brought up as part of the proposed fix for CRUNCH-272 and on the mailing list[1], a set
of jobs kicked off due to a materialize() call will not be tracked as part of the Pipeline's
stage results returned by the PipelineResult.
> [1] -

This message was sent by Atlassian JIRA

View raw message