On Sun, Mar 20, 2016 at 10:19 AM, Ashutosh Kumar <= ashutosh.di= scuss@gmail.com> wrote:

I have setup a 3 node YARN based cluster on EC= 2. I am running flink in cluster mode. I added these lines in core-site.xml=

<= configuration><= /span>

<property>
<name>fs.s3n.awsAccessKeyId</name>
<value>accesskey</value>
</property>

<property<= /span>>
= <name>fs= .s3n.awsSecretAccessKey&l= t;/name>
<value>secret key</value>
</property>
<property>
<name>fs.s3n.impl</name>
<value>org.apache.h= adoop.fs.s3native.NativeS= 3FileSystem</value>
<= ;/property>
= </configuration>
Also I added this line in conf/flink-con= f.yaml
=C2=A0=C2=A0 fs.hdfs.hadoopconf: /usr/local/hadoop/etc/hadoop

However I am getting class not found error while accessing s= 3 through s3n. I am using flink 1.0.0.

Caused by: org.apache.flink.r= untime.JobException: Creating the input splits caused an error: java.lang.R= untimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.= fs.s3native.NativeS3FileSystem not found
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.= <init>(ExecutionJobVertex.java:172)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 at org.apache.flink.runtime.executiongraph.ExecutionGraph.atta= chJobGraph(ExecutionGraph.java:696)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1= 023)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ... 25 more
Caused by= : java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNo= tFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileSystem not= found
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.c= onf.Configuration.getClass(Configuration.java:2227)
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.fs.hdfs.HadoopFileSyst= em.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.flink.core.fs.FileSys= tem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352)
=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.flink.core.fs.FileSystem= .get(FileSystem.java:280)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at = org.apache.flink.core.fs.Path.getFileSystem(Path.java:311)
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.flink.api.common.io.FileInputF= ormat.createInputSplits(FileInputFormat.java:450)
=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 at org.apache.flink.api.common.io.FileInputFormat.cre= ateInputSplits(FileInputFormat.java:57)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.= <init>(ExecutionJobVertex.java:156)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 ... 27 more
Caused by: java.lang.RuntimeException: java.lan= g.ClassNotFoundException: Class org.apache.hadoop.fs.s3native.NativeS3FileS= ystem not found
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache= .hadoop.conf.Configuration.getClass(Configuration.java:2195)
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.conf.Configuration.g= etClass(Configuration.java:2219)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 ... 34 more
Caused by: java.lang.ClassNotFoundException: Class org.a= pache.hadoop.fs.s3native.NativeS3FileSystem not found
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.conf.Configuration.getClassBy= Name(Configuration.java:2101)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)<= br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ... 35 more

= Thanks
Ashutosh