flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Kumar <ashutosh.disc...@gmail.com>
Subject Re: Access to S3 from YARN on EC2
Date Mon, 21 Mar 2016 04:56:01 GMT
Hi Tim,
I have this dependency in my pom file . This jar is present in my jar with
dependencies. I exploded the jar and checked it. The class
NativeS3FileSystem.class  is present there.

Thanks
Ashutosh


On Mon, Mar 21, 2016 at 7:20 AM, Timothy Farkas <
timothytiborfarkas@gmail.com> wrote:

> Hi Ashutosh,
>
> I believe you need to add the hadoop-aws jar to your project.
>
> http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.6.0
>
> Thanks,
> Tim
>
> On Sun, Mar 20, 2016 at 9:39 AM, Ashutosh Kumar <
> ashutosh.discuss@gmail.com> wrote:
>
>> Do I need to add some jars in lib ?
>>
>> Thanks
>> Ashutosh
>>
>> On Sun, Mar 20, 2016 at 4:30 PM, Ashutosh Kumar <
>> ashutosh.discuss@gmail.com> wrote:
>>
>>> It is not there.
>>>
>>> Thanks
>>> Ashutosh
>>>
>>> On Sun, Mar 20, 2016 at 2:58 PM, Robert Metzger <rmetzger@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> did you check if the "org.apache.hadoop.fs.s3native.NativeS3FileSystem"
>>>> class is in the flink-dist.jar in the lib/ folder?
>>>>
>>>>
>>>> On Sun, Mar 20, 2016 at 10:19 AM, Ashutosh Kumar <
>>>> ashutosh.discuss@gmail.com> wrote:
>>>>
>>>>> I have setup a 3 node YARN based cluster on EC2. I am running flink in
>>>>> cluster mode. I added these lines in core-site.xml
>>>>>
>>>>> <configuration>
>>>>>
>>>>>     <property>
>>>>>         <name>fs.s3n.awsAccessKeyId</name>
>>>>>         <value>accesskey</value>
>>>>>     </property>
>>>>>
>>>>>     <property>
>>>>>         <name>fs.s3n.awsSecretAccessKey</name>
>>>>>         <value>secret key</value>
>>>>>     </property>
>>>>>     <property>
>>>>>         <name>fs.s3n.impl</name>
>>>>>         <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
>>>>>     </property>
>>>>> </configuration>
>>>>>
>>>>> Also I added this line in conf/flink-conf.yaml
>>>>>    fs.hdfs.hadoopconf: /usr/local/hadoop/etc/hadoop
>>>>>
>>>>>
>>>>> However I am getting class not found error while accessing s3 through
>>>>> s3n. I am using flink 1.0.0.
>>>>>
>>>>> Caused by: org.apache.flink.runtime.JobException: Creating the input
>>>>> splits caused an error: java.lang.RuntimeException:
>>>>> java.lang.ClassNotFoundException: Class
>>>>> org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
>>>>>         at
>>>>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:172)
>>>>>         at
>>>>> org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:696)
>>>>>         at org.apache.flink.runtime.jobmanager.JobManager.org
>>>>> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1023)
>>>>>         ... 25 more
>>>>> Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> java.lang.ClassNotFoundException: Class
>>>>> org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
>>>>>         at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
>>>>>         at
>>>>> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
>>>>>         at
>>>>> org.apache.flink.core.fs.FileSystem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352)
>>>>>         at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:280)
>>>>>         at org.apache.flink.core.fs.Path.getFileSystem(Path.java:311)
>>>>>         at
>>>>> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:450)
>>>>>         at
>>>>> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:57)
>>>>>         at
>>>>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:156)
>>>>>         ... 27 more
>>>>> Caused by: java.lang.RuntimeException:
>>>>> java.lang.ClassNotFoundException: Class
>>>>> org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
>>>>>         at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
>>>>>         at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)
>>>>>         ... 34 more
>>>>> Caused by: java.lang.ClassNotFoundException: Class
>>>>> org.apache.hadoop.fs.s3native.NativeS3FileSystem not found
>>>>>         at
>>>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
>>>>>         at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
>>>>>         ... 35 more
>>>>>
>>>>>
>>>>> Thanks
>>>>> Ashutosh
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message