hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yiqun Lin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm
Date Mon, 06 Nov 2017 12:33:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240216#comment-16240216

Yiqun Lin commented on HDFS-12443:

I noticed there is already some commit (the commit of HDFS-10528) in trunk even if this warning
I think this shouldn't be a serious warning. So I have committed this to the feature branch.
Thanks for your patient reviews, [~cheersyang]!

> Ozone: Improve SCM block deletion throttling algorithm 
> -------------------------------------------------------
>                 Key: HDFS-12443
>                 URL: https://issues.apache.org/jira/browse/HDFS-12443
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone, scm
>            Reporter: Weiwei Yang
>            Assignee: Yiqun Lin
>              Labels: OzonePostMerge
>         Attachments: HDFS-12443-HDFS-7240.001.patch, HDFS-12443-HDFS-7240.002.patch,
HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.003.patch, HDFS-12443-HDFS-7240.004.patch,
HDFS-12443-HDFS-7240.005.patch, HDFS-12443-SCM-blockdeletion-throttle.pdf
> Currently SCM scans delLog to send deletion transactions to datanode periodically, the
throttling algorithm is simple, it scans at most {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by
default 50) at a time. This is non-optimal, worst case it might cache 50 TXs for 50 different
DNs so each DN will only get 1 TX to proceed in an interval, this will make the deletion slow.
An improvement to this is to make this throttling by datanode, e.g 50 TXs per datanode per

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message