Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (nike.apache.org: domain of stoffe@gmail.com designates
 209.85.212.176 as permitted sender)
MIME-Version: 1.0
Date: Wed, 11 Jun 2014 16:23:07 +0200
Message-ID: 
 <CADSBNAiiM+VHnWGGJeP9NBqY1Xz8w1BFbiFrysg=qxgWq+ZT=Q@mail.gmail.com>
Subject: CDH5
From: =?UTF-8?Q?Kristoffer_Sj=C3=B6gren?= <stoffe@gmail.com>
To: user@crunch.apache.org
Content-Type: text/plain; charset=UTF-8

Hi

Im trying out Crunch on YARN on CDH5 (0.9.0-cdh5.0.0) and get some
errors when trying to materialize results (see below). The job itself
is super simple.

PCollection<String> lines = pipeline.read(new TextFileSource<String>(
    new Path("hdfs://....log"), Writables.strings()));

lines = lines.parallelDo(new DoFn<String, String>() {
  @Override
  public void process(String s, Emitter<String> e) {
    e.emit(s);
  }
}, Writables.strings());

for (String line : lines.materialize()) {
  System.out.println(line);
}


Seems like there's some kind of sync issue here because I can see the
"correct" tmp dir in hdfs. Note that the p index is "p2" in hdfs while
the client looks for "p1".

-rw-r--r--   1 kristoffersjogren supergroup       1748 2014-06-11
15:36 /tmp/crunch-134908575/p2/MAP
drwxr-xr-x   - kristoffersjogren supergroup          0 2014-06-11
15:36 /tmp/crunch-134908575/p2/output
-rw-r--r--   1 kristoffersjogren supergroup          0 2014-06-11
15:36 /tmp/crunch-134908575/p2/output/_SUCCESS
-rw-r--r--   1 kristoffersjogren supergroup   42898831 2014-06-11
15:36 /tmp/crunch-134908575/p2/output/out0-m-00000
-rw-r--r--   1 kristoffersjogren supergroup          0 2014-06-11
15:36 /tmp/crunch-134908575/p2/output/part-m-00000


If I try to write directly to HDFS using the following, the job finish
successfully, but nothing is written instead?

pipeline.write(lines, new TextFileSourceTarget<String>("/user/stoffe",
Writables.strings()), WriteMode.OVERWRITE);


Any ideas of what might go wrong?

Cheers,
-Kristoffer


Exception in thread "main" java.lang.RuntimeException:
org.apache.crunch.CrunchRuntimeException: java.io.IOException: No
files found to materialize at: /tmp/crunch-1611606737/p1
at mapred.CrunchJob.<init>(CrunchJob.java:36)
at mapred.tempjobs.DownloadFiles.<init>(DownloadFiles.java:16)
at mapred.tempjobs.DownloadFiles.main(DownloadFiles.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: org.apache.crunch.CrunchRuntimeException:
java.io.IOException: No files found to materialize at:
/tmp/crunch-1611606737/p1
at org.apache.crunch.materialize.MaterializableIterable.materialize(MaterializableIterable.java:79)
at org.apache.crunch.materialize.MaterializableIterable.iterator(MaterializableIterable.java:69)
at mapred.tempjobs.DownloadFiles.run(DownloadFiles.java:37)
at mapred.CrunchJob.run(CrunchJob.java:96)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at mapred.CrunchJob.<init>(CrunchJob.java:34)
... 7 more
Caused by: java.io.IOException: No files found to materialize at:
/tmp/crunch-1611606737/p1
at org.apache.crunch.io.CompositePathIterable.create(CompositePathIterable.java:49)
at org.apache.crunch.io.impl.FileSourceImpl.read(FileSourceImpl.java:136)
at org.apache.crunch.io.seq.SeqFileSource.read(SeqFileSource.java:43)
at org.apache.crunch.io.impl.ReadableSourcePathTargetImpl.read(ReadableSourcePathTargetImpl.java:37)
at org.apache.crunch.materialize.MaterializableIterable.materialize(MaterializableIterable.java:76)
... 12 more