flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Access to S3 from YARN on EC2
Date Sun, 20 Mar 2016 09:28:27 GMT
Hi,

did you check if the "org.apache.hadoop.fs.s3native.NativeS3FileSystem"
class is in the flink-dist.jar in the lib/ folder?


On Sun, Mar 20, 2016 at 10:19 AM, Ashutosh Kumar <ashutosh.discuss@gmail.com
> wrote:

> I have setup a 3 node YARN based cluster on EC2. I am running flink in
> cluster mode. I added these lines in core-site.xml
>
> <configuration>
>
>     <property>
>         <name>fs.s3n.awsAccessKeyId</name>
>         <value>accesskey</value>
>     </property>
>
>     <property>
>         <name>fs.s3n.awsSecretAccessKey</name>
>         <value>secret key</value>
>     </property>
>     <property>
>         <name>fs.s3n.impl</name>
>         <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
>     </property>
> </configuration>
>
> Also I added this line in conf/flink-conf.yaml
>    fs.hdfs.hadoopconf: /usr/local/hadoop/etc/hadoop
>
>
> However I am getting class not found error while accessing s3 through s3n.
> I am using flink 1.0.0.
>
> Caused by: org.apache.flink.runtime.JobException: Creating the input
> splits caused an error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
>         at
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:172)
>         at
> org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:696)
>         at org.apache.flink.runtime.jobmanager.JobManager.org
> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1023)
>         ... 25 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
>         at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
>         at
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
>         at
> org.apache.flink.core.fs.FileSystem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352)
>         at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:280)
>         at org.apache.flink.core.fs.Path.getFileSystem(Path.java:311)
>         at
> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:450)
>         at
> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:57)
>         at
> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
>         ... 27 more
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
>         at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
>         at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)
>         ... 34 more
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
>         at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
>         at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
>         ... 35 more
>
>
> Thanks
> Ashutosh
>
>

Mime
View raw message