hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
Date Tue, 29 Aug 2017 01:37:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144634#comment-16144634

Weiwei Yang commented on HDFS-12354:

Hi [~linyiqun]

Thanks for posting the first patch, it looks very good overall. However, I have a different
opinion on the interface signature. Currently it is

List<ContainerData> chooseContainerForBlockDeletion(String prefix, long count,
    String prevKey, Map<String, ContainerStatus> candidateContainers) throws StorageContainerException;

Few comments
# Can we just maintain {{prefix}} and {{prevKey}} internally in an implementation class (if
necessary) instead of exposing them via arguments? Want to keep the method simple to call.
# Can we use {{int}} type count?

NIT in ozone-default.xml, can we add some more doc here

Datanode selects a number of containers to process block deletion in a certain interval defined
by _ozone.block.deleting.service.interval.ms_, the number of containers to process in each
interval is defined by _ozone.block.deleting.container.limit.per.interval_. This property
is used to configure the policy applied while selecting containers, _org.apache.hadoop.ozone.container.common.impl.RandomContainerDeletionChoosingPolicy_
implements a simply random policy that to return a random list of containers.


> Ozone: Shuffle container list for datanode BlockDeletingService
> ---------------------------------------------------------------
>                 Key: HDFS-12354
>                 URL: https://issues.apache.org/jira/browse/HDFS-12354
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, ozone
>            Reporter: Weiwei Yang
>            Assignee: Yiqun Lin
>         Attachments: HDFS-12354-HDFS-7240.001.patch
> {{BlockDeletingService}} is a per-datanode container block deleting service takes in
charge of the "real" deletion of ozone blocks. It spawns a worker thread per container and
delete blocks/chunks from disk as background threads. The number of threads currently is throttled
by {{ozone.block.deleting.container.limit.per.interval}}, but there is a potential problem.
Containers are sorted so it always fetch same of containers, we need to fix this by creating
an API in {{ContainerManagerImpl}} to get a shuffled list of containers.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message