crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-509) Crunch with Spark doesn't name all outputs
Date Thu, 09 Apr 2015 02:04:13 GMT


Micah Whitacre updated CRUNCH-509:
    Attachment: CRUNCH-509.patch

Still working on solution for this.  The change to add name support is pretty simple.  The
downstream effect however is that all calls to materialize the output (which is what we do
in the IT for Spark) fail because it cannot find the files.

4500 [Thread-29] INFO  org.apache.spark.scheduler.DAGScheduler  - Job 0 finished: saveAsNewAPIHadoopFile
at, took 0.874098 s
15/04/08 20:57:48 INFO DAGScheduler: Job 0 finished: saveAsNewAPIHadoopFile at,
took 0.874098 s
4573 [main] INFO  - Could not read avro file
at path: file:/tmp/crunch-109470525/p1/part-r-00000 Not a data file.
	at org.apache.avro.file.DataFileStream.initialize(
	at org.apache.avro.file.DataFileReader.<init>(
	at org.apache.crunch.materialize.pobject.FirstElementPObject.process(
	at org.apache.crunch.materialize.pobject.PObjectImpl.getValue(
	at org.apache.crunch.SparkPageRankIT.testAvroReflects(

One of the behavior changes I noticed is that when ran without a name, the job produces files
that are named, part-r-00000.avro.  When we add the name we are now getting files without
the file extension.  I believe this might be related to it not being able to detect the files
as containing data but I haven't found in the code where that extension might be getting dropped.

> Crunch with Spark doesn't name all outputs
> ------------------------------------------
>                 Key: CRUNCH-509
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.11.0
>            Reporter: Micah Whitacre
>            Assignee: Josh Wills
>             Fix For: 0.12.0
>         Attachments: CRUNCH-509.patch
> Crunch currently does not "name" all outputs when running with a SparkPipeline.  This
becomes a problem as some Targets (based on CRUNCH-82) have coded in checked to ensure that
the name must be populated.  Specifically the implementation I'm running into issues with
is the Kite DatasetTarget[2].
> Need to read up a bit on context to see if it is a Crunch/Kite issue or where it is easiest/correct
to fix.  [~jwills] or [~tomwhite] feedback would be welcome.
> [1] -
> [2] -

This message was sent by Atlassian JIRA

View raw message