hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Inigo Goiri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12636) Catch exception in loadFileSystems()
Date Mon, 14 Dec 2015 01:21:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055248#comment-15055248
] 

Inigo Goiri commented on HADOOP-12636:
--------------------------------------

One example is when S3AFileSystem is in the path and we don't have the AWS SDK in the path.
Even when we are trying to load HDFS, we get this fatal error:

java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem
could not be instantiated
                at java.util.ServiceLoader.fail(ServiceLoader.java:224)
                at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
                at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
                at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
                at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2617)
                at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2633)
                at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2650)
                at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
                at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2689)
                at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2671)
                at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
                at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:342)
                at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2284)
                at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2182)
                at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2229)
                at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2409)
                at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2433)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentialsProvider
                at java.lang.Class.getDeclaredConstructors0(Native Method)
                at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532)
                at java.lang.Class.getConstructor0(Class.java:2842)
                at java.lang.Class.newInstance(Class.java:345)
                at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
                ... 14 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider
                at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
                at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
                at java.security.AccessController.doPrivileged(Native Method)
                at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
                ... 19 more

I propose to capture these exception in loadFilesystems and when the user actually tries to
use the failed Filesystem will get the proper error.

> Catch exception in loadFileSystems() 
> -------------------------------------
>
>                 Key: HADOOP-12636
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12636
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.6.2
>            Reporter: Inigo Goiri
>            Assignee: Inigo Goiri
>             Fix For: 2.6.3
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> loadFileSystems() loads all the Filesystems in the path. However, some Filesystems cannot
be initialized. There is no point on failing the startup because a Filesystem that won't be
used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message