flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject writeAsCsv on HDFS
Date Thu, 25 Jun 2015 12:50:08 GMT
Hi to all,
I'm experiencing some problem in writing a file as csv on HDFS with flink
0.9.0.
The code I use is
  myDataset.writeAsCsv(new Path("hdfs:///tmp", "myFile.csv").toString());

If I run the job from Eclipse everything works fine but when I deploy the
job on the cluster (cloudera 5.1.3) I obtain the following exception:

Caused by: java.io.IOException: The given HDFS file URI
(hdfs:///tmp/myFile.csv) did not describe the HDFS NameNode. The attempt to
use a default HDFS configuration, as specified in the 'fs.hdfs.hdfsdefault'
or 'fs.hdfs.hdfssite' config parameter failed due to the following problem:
Either no default file system was registered, or the provided configuration
contains no valid authority component (fs.default.name or fs.defaultFS)
describing the (hdfs namenode) host and port.
at
org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:291)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:258)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:309)
at
org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:273)
at
org.apache.flink.runtime.jobgraph.OutputFormatVertex.initializeOnMaster(OutputFormatVertex.java:84)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$org$apache$flink$runtime$jobmanager$JobManager$$submitJob$4.apply(JobManager.scala:520)
... 25 more

The core-site.xml is present in the fat jar and contains the property

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://myServerX:8020</value>
  </property>

I compiled flink with the following command:

 mvn clean  install -Dhadoop.version=2.3.0-cdh5.1.3
-Dhbase.version=0.98.1-cdh5.1.3 -Dhadoop.core.version=2.3.0-mr1-cdh5.1.3
-DskipTests -Pvendor-repos

How can I fix that?

Best,
Flavio

Mime
View raw message