hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maciej Arciuch (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-7272) Concurrency problem in Dfsck/URLConnectionFactory classes
Date Tue, 21 Oct 2014 15:16:33 GMT
Maciej Arciuch created HDFS-7272:

             Summary: Concurrency problem in Dfsck/URLConnectionFactory classes
                 Key: HDFS-7272
                 URL: https://issues.apache.org/jira/browse/HDFS-7272
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 2.3.0
            Reporter: Maciej Arciuch
            Priority: Minor


I think I've found a concurrency issue in the URLConnectionFactory class used by the Dfsck

The problem is that when multiple instances of new Dfsck run in the same JVM, 401 errors occur.
I know that this class in usually run in different command-line processes, but if I call programatically
Dfsck.doWork() on multiple instances in parallel I get strange, random auth errors. Earlier
versions worked fine.

So, I took a look at the code and that's I found:

In the earlier versions of the Dfsck the connection was instatiated using SecurityUtil.openSecureHttpConnection(path),
which worked just fine. New versions use URLConnectionFactory.

Quick links to grepcode:

old, correct code:

new code:

The difference is that the old SecurityUtil always passed null Authenticator to the AuthenticatedURL
constructor, which, if case of null argument, instantiated a new Authenticator instance using
the default instantiator class. *So, there was new Authenticator instance created for each
call. The URLConnectorFactory passes a non-null, static Authenticator, which is stateful and
not thread-safe.*

old code:

new code:

Can anyone confirm and perhaps fix this? Best regards, Maciej

This message was sent by Atlassian JIRA

View raw message