flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Cuquemelle (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-9245) Can't create a BucketingSink with a provided Configuration if no hadoop defaults
Date Tue, 24 Apr 2018 07:50:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-9245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Julien Cuquemelle updated FLINK-9245:
-------------------------------------
    Description: 
We build Integration tests using this kind of code: 
{code:java}
val bucketingSink = new BucketingSink[Row]("hdfs:///user/$USER)}/application_name/")
bucketingSink.setFSConfig(hadoopRule.getConfiguration.hdfs) bucketingSink.setBucketer(new
DateTimeBucketer[Row]("yyyy-MM-dd--HHmm")) outpuStream.addSink(bucketingSink)
{code}
Here, the hadoopRule is providing a valid hdfs config that should allows this kind of code
to run on a machine with no HADOOP_HOME or HADOOP_CONF_DIR set up, like a developper workstation
or a Jenkins slave.

When running this code on such a machine, the .createHadoopFileSystem(...) fails with 
{noformat}
The given file system URI (hdfs:///user/$USER/application_name/) did not describe the authority

at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401){noformat}
because it tries to instantiate the fileSystem from a default configuration in .getUnguardedFileSystem()
; as the default conf doesn't exist, the default filesystem resolves to "file:///" and the
checks of the consistency of the URI fails because no authority can be found So the whole
filesystem creation fails before actually trying to use the provided config.

  was:
We build Integration tests using this kind of code: 
{code:java}
val bucketingSink = new BucketingSink[Row](s"hdfs:///user/${System.getenv("USER")}/application_name/")
bucketingSink.setFSConfig(hadoopRule.getConfiguration.hdfs) bucketingSink.setBucketer(new
DateTimeBucketer[Row]("yyyy-MM-dd--HHmm")) outpuStream.addSink(bucketingSink)
{code}
Here, the hadoopRule is providing a valid hdfs config that should allows this kind of code
to run on a machine with no HADOOP_HOME or HADOOP_CONF_DIR set up, like a developper workstation
or a Jenkins slave.

When running this code on such a machine, the .createHadoopFileSystem(...) fails with 
{noformat}
The given file system URI (hdfs:///user/$USER/application_name/) did not describe the authority

at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401){noformat}
because it tries to instantiate the fileSystem from a default configuration in .getUnguardedFileSystem()
; as the default conf doesn't exist, the default filesystem resolves to "file:///" and the
checks of the consistency of the URI fails because no authority can be found So the whole
filesystem creation fails before actually trying to use the provided config.


> Can't create a BucketingSink with a provided Configuration if no hadoop defaults
> --------------------------------------------------------------------------------
>
>                 Key: FLINK-9245
>                 URL: https://issues.apache.org/jira/browse/FLINK-9245
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>    Affects Versions: 1.4.2
>            Reporter: Julien Cuquemelle
>            Priority: Minor
>
> We build Integration tests using this kind of code: 
> {code:java}
> val bucketingSink = new BucketingSink[Row]("hdfs:///user/$USER)}/application_name/")
> bucketingSink.setFSConfig(hadoopRule.getConfiguration.hdfs) bucketingSink.setBucketer(new
DateTimeBucketer[Row]("yyyy-MM-dd--HHmm")) outpuStream.addSink(bucketingSink)
> {code}
> Here, the hadoopRule is providing a valid hdfs config that should allows this kind of
code to run on a machine with no HADOOP_HOME or HADOOP_CONF_DIR set up, like a developper
workstation or a Jenkins slave.
> When running this code on such a machine, the .createHadoopFileSystem(...) fails with 
> {noformat}
> The given file system URI (hdfs:///user/$USER/application_name/) did not describe the
authority 
> at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
> at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401){noformat}
> because it tries to instantiate the fileSystem from a default configuration in .getUnguardedFileSystem()
; as the default conf doesn't exist, the default filesystem resolves to "file:///" and the
checks of the consistency of the URI fails because no authority can be found So the whole
filesystem creation fails before actually trying to use the provided config.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message