flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yan Yan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-10526) Hadoop FileSystem not initialized properly on Yarn
Date Wed, 10 Oct 2018 23:41:00 GMT
Yan Yan created FLINK-10526:

             Summary: Hadoop FileSystem not initialized properly on Yarn
                 Key: FLINK-10526
                 URL: https://issues.apache.org/jira/browse/FLINK-10526
             Project: Flink
          Issue Type: Bug
          Components: YARN
    Affects Versions: 1.4.0
            Reporter: Yan Yan

When we were bumping Flink from 1.3.2 to 1.4.0, we noticed the HadoopFileSystem no longer
reads flinkConfig. 

This prevents users to pass in custom HDFS configuration (e.g. core-site.xml) via _fs.hdfs.hadoopconf_.

Specially it is due to the cached _HadoopFsFactory_ is initialized with an dummy Configuration
which prevents future _flinkConfig_ getting passed in ([[code|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFsFactory.java#L84]|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFsFactory.java#L84]).

We have verified by calling _FileSystem.initialize(flinkConfig)_ explicitly in _YarnApplicationMasterRunner.run_ solve
the issue.


This message was sent by Atlassian JIRA

View raw message