hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9478) Fix race conditions during the initialization of Configuration related to deprecatedKeyMap
Date Sat, 09 Nov 2013 05:23:19 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13818019#comment-13818019
] 

Colin Patrick McCabe commented on HADOOP-9478:
----------------------------------------------

Why don't you file an MR JIRA with a description of the problems you are having and how to
reproduce them so that the community can help with this process?  If this JIRA turns out to
be the issue we can always link the JIRAs, like we did with so many other Configuration issues.

> Fix race conditions during the initialization of Configuration related to deprecatedKeyMap
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-9478
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9478
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 2.0.0-alpha
>         Environment: OS:
> CentOS release 6.3 (Final)
> JDK:
> java version "1.6.0_27"
> Java(TM) SE Runtime Environment (build 1.6.0_27-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)
> Hadoop:
> hadoop-2.0.0-cdh4.1.3/hadoop-2.0.0-cdh4.2.0
> Security:
> Kerberos
>            Reporter: Dongyong Wang
>            Assignee: Colin Patrick McCabe
>             Fix For: 2.2.1
>
>         Attachments: HADOOP-9478.001.patch, HADOOP-9478.002.patch, HADOOP-9478.003.patch,
HADOOP-9478.004.patch, HADOOP-9478.005.patch, hadoop-9478-1.patch, hadoop-9478-2.patch
>
>
> When we lanuch the client appliation which use kerberos security,the FileSystem can't
be create because the exception ' java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.security.SecurityUtil'.
> I check the exception stack trace,it maybe caused by the unsafe get operation of the
deprecatedKeyMap which used by the org.apache.hadoop.conf.Configuration.
> So I write a simple test case:
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.hdfs.HdfsConfiguration;
> public class HTest {
>     public static void main(String[] args) throws Exception {
>         Configuration conf = new Configuration();
>         conf.addResource("core-site.xml");
>         conf.addResource("hdfs-site.xml");
>         FileSystem fileSystem = FileSystem.get(conf);
>         System.out.println(fileSystem);
>         System.exit(0);
>     }
> }
> Then I launch this test case many times,the following exception is thrown:
> Exception in thread "TGT Renewer for XXX" java.lang.ExceptionInInitializerError
>      at org.apache.hadoop.security.UserGroupInformation.getTGT(UserGroupInformation.java:719)
>      at org.apache.hadoop.security.UserGroupInformation.access$1100(UserGroupInformation.java:77)
>      at org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:746)
>      at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 16
>      at java.util.HashMap.getEntry(HashMap.java:345)
>      at java.util.HashMap.containsKey(HashMap.java:335)
>      at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1989)
>      at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1867)
>      at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1785)
>      at org.apache.hadoop.conf.Configuration.get(Configuration.java:712)
>      at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:731)
>      at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1047)
>      at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:76)
>      ... 4 more
> Exception in thread "main" java.io.IOException: Couldn't create proxy provider class
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>      at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:453)
>      at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:133)
>      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:436)
>      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:403)
>      at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:125)
>      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2262)
>      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
>      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2296)
>      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2278)
>      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
>      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)
>      at HTest.main(HTest.java:11)
> Caused by: java.lang.reflect.InvocationTargetException
>      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>      at org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider(NameNodeProxies.java:442)
>      ... 11 more
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.SecurityUtil
>      at org.apache.hadoop.net.NetUtils.createSocketAddrForHost(NetUtils.java:231)
>      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:211)
>      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:159)
>      at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
>      at org.apache.hadoop.hdfs.DFSUtil.getAddressesForNameserviceId(DFSUtil.java:452)
>      at org.apache.hadoop.hdfs.DFSUtil.getAddresses(DFSUtil.java:434)
>      at org.apache.hadoop.hdfs.DFSUtil.getHaNnRpcAddresses(DFSUtil.java:496)
>      at org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.<init>(ConfiguredFailoverProxyProvider.java:88)
>      ... 16 more
> If the HashMap used at multi-thread enviroment,not only the put operation be synchronized,the
get operation(eg. containKey) should be synchronzied too.
> The simple solution is trigger the init of SecurityUtil before creating the FileSystem,but
I think it's should be synchronized for get of deprecatedKeyMap.
> Thanks. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message