crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-509) Crunch with Spark doesn't name all outputs
Date Wed, 08 Apr 2015 15:21:12 GMT


Micah Whitacre commented on CRUNCH-509:

[~tomwhite] unfortunately just naming things doesn't fix the issue.  The outputs aren't in
the location necessary for things like materialize to work with Spark.

[~jwills] Thanks for the hint.  I was going to try taking a stab at this change today and
will play around.

> Crunch with Spark doesn't name all outputs
> ------------------------------------------
>                 Key: CRUNCH-509
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.11.0
>            Reporter: Micah Whitacre
>            Assignee: Josh Wills
>             Fix For: 0.12.0
> Crunch currently does not "name" all outputs when running with a SparkPipeline.  This
becomes a problem as some Targets (based on CRUNCH-82) have coded in checked to ensure that
the name must be populated.  Specifically the implementation I'm running into issues with
is the Kite DatasetTarget[2].
> Need to read up a bit on context to see if it is a Crunch/Kite issue or where it is easiest/correct
to fix.  [~jwills] or [~tomwhite] feedback would be welcome.
> [1] -
> [2] -

This message was sent by Atlassian JIRA

View raw message