hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maciej Arciuch (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-7272) Concurrency problem in Dfsck/URLConnectionFactory classes
Date Tue, 21 Oct 2014 15:16:33 GMT
Maciej Arciuch created HDFS-7272:
------------------------------------

             Summary: Concurrency problem in Dfsck/URLConnectionFactory classes
                 Key: HDFS-7272
                 URL: https://issues.apache.org/jira/browse/HDFS-7272
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 2.3.0
            Reporter: Maciej Arciuch
            Priority: Minor


Hi,

I think I've found a concurrency issue in the URLConnectionFactory class used by the Dfsck
class. 

The problem is that when multiple instances of new Dfsck run in the same JVM, 401 errors occur.
I know that this class in usually run in different command-line processes, but if I call programatically
Dfsck.doWork() on multiple instances in parallel I get strange, random auth errors. Earlier
versions worked fine.

So, I took a look at the code and that's I found:

In the earlier versions of the Dfsck the connection was instatiated using SecurityUtil.openSecureHttpConnection(path),
which worked just fine. New versions use URLConnectionFactory.

Quick links to grepcode:

old, correct code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-hdfs/2.0.0-cdh4.2.1/org/apache/hadoop/hdfs/tools/DFSck.java/#161

new code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-hdfs/2.3.0-cdh5.1.2/org/apache/hadoop/hdfs/tools/DFSck.java/#174

The difference is that the old SecurityUtil always passed null Authenticator to the AuthenticatedURL
constructor, which, if case of null argument, instantiated a new Authenticator instance using
the default instantiator class. *So, there was new Authenticator instance created for each
call. The URLConnectorFactory passes a non-null, static Authenticator, which is stateful and
not thread-safe.*

old code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-common/2.0.0-cdh4.2.1/org/apache/hadoop/security/SecurityUtil.java#508

new code:
http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop/hadoop-hdfs/2.3.0-cdh5.1.2/org/apache/hadoop/hdfs/web/URLConnectionFactory.java#164

Can anyone confirm and perhaps fix this? Best regards, Maciej



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message