hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
Date Fri, 08 Sep 2017 22:58:01 GMT
Lei (Eddy) Xu created HDFS-12412:

             Summary: Remove ErasureCodingWorker.stripedReadPool
                 Key: HDFS-12412
                 URL: https://issues.apache.org/jira/browse/HDFS-12412
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: erasure-coding
    Affects Versions: 3.0.0-alpha3
            Reporter: Lei (Eddy) Xu
            Assignee: Lei (Eddy) Xu

In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule the EC recovery
tasks, while uses {{stripedReadPool}} for the reader threads in each recovery task.  We only
need one of them to throttle the speed of recovery process, because each EC recovery task
has a fix number of source readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044,
the speed of EC recovery can be throttled by {{strippedReconstructionPool}} with {{xmitsInProgress}}.

Moreover, keeping {{stripedReadPool}} makes customer difficult to understand and calculate
the right balance between {{dfs.datanode.ec.reconstruction.stripedread.threads}}, {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}}
and {{maxReplicationStreams}}.  For example, a small {{stripread.threads}} (comparing to which
{{reconstruction.threads.size}} implies), will unnecessarily limit the speed of recovery,
which leads to larger MTTR. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message