Return-Path: <dev-return-12493-archive-asf-public=cust-asf.ponee.io@flink.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 0483B200B32
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Thu, 23 Jun 2016 22:08:18 +0200 (CEST)
Received: by cust-asf.ponee.io (Postfix)
	id 03220160A68; Thu, 23 Jun 2016 20:08:18 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 4B566160A35
	for <archive-asf-public@cust-asf.ponee.io>; Thu, 23 Jun 2016 22:08:17 +0200 (CEST)
Received: (qmail 54505 invoked by uid 500); 23 Jun 2016 20:08:16 -0000
Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@flink.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@flink.apache.org>
List-Post: <mailto:dev@flink.apache.org>
List-Id: <dev.flink.apache.org>
Reply-To: dev@flink.apache.org
Delivered-To: mailing list dev@flink.apache.org
Received: (qmail 54493 invoked by uid 99); 23 Jun 2016 20:08:16 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2016 20:08:16 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 1C90B2C033A
	for <dev@flink.apache.org>; Thu, 23 Jun 2016 20:08:16 +0000 (UTC)
Date: Thu, 23 Jun 2016 20:08:16 +0000 (UTC)
From: "Josh Forman-Gornall (JIRA)" <jira@apache.org>
To: dev@flink.apache.org
Message-ID: <JIRA.12982446.1466712467000.13794.1466712496112@Atlassian.JIRA>
In-Reply-To: <JIRA.12982446.1466712467000@Atlassian.JIRA>
References: <JIRA.12982446.1466712467000@Atlassian.JIRA> <JIRA.12982446.1466712467796@arcas>
Subject: [jira] [Created] (FLINK-4115) FsStateBackend filesystem
 verification can cause classpath exceptions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Thu, 23 Jun 2016 20:08:18 -0000

Josh Forman-Gornall created FLINK-4115:
------------------------------------------

             Summary: FsStateBackend filesystem verification can cause classpath exceptions
                 Key: FLINK-4115
                 URL: https://issues.apache.org/jira/browse/FLINK-4115
             Project: Flink
          Issue Type: Bug
          Components: Core
    Affects Versions: 1.1.0
            Reporter: Josh Forman-Gornall
            Priority: Minor


In the constructor of FsStateBackend, the FileSystem for the checkpoint directory is initialised and it is verified that the checkpoint path exists. This verification happens in the Flink client program when submitting a job and can cause classpath issues if classes required to access the file system are not available in the client's classpath.

For example, if we run Flink on YARN over AWS EMR using RocksDBStateBackend and an s3:// checkpoint directory, we get the below ClassNotFoundException. This is because the jars needed to use the EMR file system are available only in the YARN context and not when submitting the job via the Flink client.

```
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
	at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
	at org.apache.flink.core.fs.FileSystem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352)
	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:280)
	at org.apache.flink.runtime.state.filesystem.FsStateBackend.validateAndNormalizeUri(FsStateBackend.java:383)
	at org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:175)
	at org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:144)
	at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:205)
```



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)