flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2842) S3FileSystem is broken
Date Fri, 09 Oct 2015 14:43:26 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950494#comment-14950494
] 

Robert Metzger commented on FLINK-2842:
---------------------------------------

It only works for file systems which have an {{fs.XXX.impl}} entry in the hadoop configuraiton.
With Hadoop2, the mechanism for loading file systems has been changed, but we can not use
it because flink-runtime has to be backwards compatible with Hadoop1.

See http://stackoverflow.com/questions/32959790/run-apache-flink-with-amazon-s3

> S3FileSystem is broken
> ----------------------
>
>                 Key: FLINK-2842
>                 URL: https://issues.apache.org/jira/browse/FLINK-2842
>             Project: Flink
>          Issue Type: Bug
>    Affects Versions: 0.9, 0.10
>            Reporter: Fabian Hueske
>            Assignee: Fabian Hueske
>            Priority: Critical
>             Fix For: 0.10
>
>
> Flink has a file system implementation for S3 which is broken. Therefore, Flink cannot
out-of-the-box read data from S3.
> Since Flink can use all Hadoop FileSystems, I propose to completely remove Flink's S3FileSystem
and rely on Hadoop's implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message