Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 41A0018895 for ; Mon, 14 Dec 2015 01:21:47 +0000 (UTC) Received: (qmail 40723 invoked by uid 500); 14 Dec 2015 01:21:46 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 40633 invoked by uid 500); 14 Dec 2015 01:21:46 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 40519 invoked by uid 99); 14 Dec 2015 01:21:46 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Dec 2015 01:21:46 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id A40582C1F6B for ; Mon, 14 Dec 2015 01:21:46 +0000 (UTC) Date: Mon, 14 Dec 2015 01:21:46 +0000 (UTC) From: "Inigo Goiri (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-12636) Catch exception in loadFileSystems() MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055248#comment-15055248 ] Inigo Goiri commented on HADOOP-12636: -------------------------------------- One example is when S3AFileSystem is in the path and we don't have the AWS SDK in the path. Even when we are trying to load HDFS, we get this fatal error: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:224) at java.util.ServiceLoader.access$100(ServiceLoader.java:181) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377) at java.util.ServiceLoader$1.next(ServiceLoader.java:445) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2617) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2633) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2650) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2689) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2671) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:342) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2284) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2182) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2229) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2409) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2433) Caused by: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentialsProvider at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532) at java.lang.Class.getConstructor0(Class.java:2842) at java.lang.Class.newInstance(Class.java:345) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373) ... 14 more Caused by: java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 19 more I propose to capture these exception in loadFilesystems and when the user actually tries to use the failed Filesystem will get the proper error. > Catch exception in loadFileSystems() > ------------------------------------- > > Key: HADOOP-12636 > URL: https://issues.apache.org/jira/browse/HADOOP-12636 > Project: Hadoop Common > Issue Type: Bug > Components: fs > Affects Versions: 2.6.2 > Reporter: Inigo Goiri > Assignee: Inigo Goiri > Fix For: 2.6.3 > > Original Estimate: 1h > Remaining Estimate: 1h > > loadFileSystems() loads all the Filesystems in the path. However, some Filesystems cannot be initialized. There is no point on failing the startup because a Filesystem that won't be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)