flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Forman-Gornall (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-4115) FsStateBackend filesystem verification can cause classpath exceptions
Date Thu, 23 Jun 2016 20:08:16 GMT
Josh Forman-Gornall created FLINK-4115:

             Summary: FsStateBackend filesystem verification can cause classpath exceptions
                 Key: FLINK-4115
                 URL: https://issues.apache.org/jira/browse/FLINK-4115
             Project: Flink
          Issue Type: Bug
          Components: Core
    Affects Versions: 1.1.0
            Reporter: Josh Forman-Gornall
            Priority: Minor

In the constructor of FsStateBackend, the FileSystem for the checkpoint directory is initialised
and it is verified that the checkpoint path exists. This verification happens in the Flink
client program when submitting a job and can cause classpath issues if classes required to
access the file system are not available in the client's classpath.

For example, if we run Flink on YARN over AWS EMR using RocksDBStateBackend and an s3:// checkpoint
directory, we get the below ClassNotFoundException. This is because the jars needed to use
the EMR file system are available only in the YARN context and not when submitting the job
via the Flink client.

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem
not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
	at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
	at org.apache.flink.core.fs.FileSystem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352)
	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:280)
	at org.apache.flink.runtime.state.filesystem.FsStateBackend.validateAndNormalizeUri(FsStateBackend.java:383)
	at org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:175)
	at org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:144)
	at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:205)

This message was sent by Atlassian JIRA

View raw message