hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Zhuge (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-11900) HEDGED_READ_THREAD_POOL should not be static
Date Mon, 29 May 2017 09:45:04 GMT
John Zhuge created HDFS-11900:

             Summary: HEDGED_READ_THREAD_POOL should not be static
                 Key: HDFS-11900
                 URL: https://issues.apache.org/jira/browse/HDFS-11900
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: hdfs-client
    Affects Versions: 2.8.0
            Reporter: John Zhuge

*Non-static* synchronized method initThreadsNumForHedgedReads can't synchronize the access
to the *static* class variable HEDGED_READ_THREAD_POOL.
  private static ThreadPoolExecutor HEDGED_READ_THREAD_POOL;
  private synchronized void initThreadsNumForHedgedReads(int num) {
2 DFS clients may update the same static variable in a race because the lock is on each DFS
client object, not on the shared DFSClient class object.

There are 2 possible fixes:
1. "Global thread pool": Change initThreadsNumForHedgedReads to static
2. "Per-client thread pool": Change HEDGED_READ_THREAD_POOL to non-static

>From the description for property {{dfs.client.hedged.read.threadpool.size}}:
    to a positive number. The threadpool size is how many threads to dedicate
    to the running of these 'hedged', concurrent reads in your client.
it seems to indicate the thread pool is per DFS client.

Let's assume we go with #1 "Global thread pool". One DFS client has the property set to 10
in its config, while the other client has the property set to 5 in its config, what is supposed
to the size of the global thread pool? 5? 10? Or 15?

The 2nd fix seems more reasonable to me.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message