hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-14657) Refine NameSystem lock usage during processing FBR
Date Sun, 21 Jul 2019 17:17:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chen Zhang updated HDFS-14657:
    Attachment: HDFS-14657.002.patch

> Refine NameSystem lock usage during processing FBR
> --------------------------------------------------
>                 Key: HDFS-14657
>                 URL: https://issues.apache.org/jira/browse/HDFS-14657
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Chen Zhang
>            Assignee: Chen Zhang
>            Priority: Major
>         Attachments: HDFS-14657-001.patch, HDFS-14657.002.patch
> The disk with 12TB capacity is very normal today, which means the FBR size is much larger
than before, Namenode holds the NameSystemLock during processing block report for each storage,
which might take quite a long time.
> On our production environment, processing large FBR usually cause a longer RPC queue
time, which impacts client latency, so we did some simple work on refining the lock usage,
which improved the p99 latency significantly.
> In our solution, BlockManager release the NameSystem write lock and request it again
for every 5000 blocks(by default) during processing FBR, with the fair lock, all the RPC request
can be processed before BlockManager re-acquire the write lock.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message