hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12354) Ozone: Shuffle container list for datanode BlockDeletingService
Date Fri, 25 Aug 2017 09:49:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16141433#comment-16141433
] 

Weiwei Yang commented on HDFS-12354:
------------------------------------

Hi [~linyiqun]

I agree, that's even better. But I am not sure if there is a neat way to figure out such top
N containers. That might involve some queries on containers DB since that's the only place
we track blocks under deletion (currently). That would be too heavy if we do this every time,
because there might be more than 4000 containers in a fully loaded datanode. Feel free to
take this over and share your thoughts if you have an idea how to achieve this. Thanks.

> Ozone: Shuffle container list for datanode BlockDeletingService
> ---------------------------------------------------------------
>
>                 Key: HDFS-12354
>                 URL: https://issues.apache.org/jira/browse/HDFS-12354
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, ozone
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>
> {{BlockDeletingService}} is a per-datanode container block deleting service takes in
charge of the "real" deletion of ozone blocks. It spawns a worker thread per container and
delete blocks/chunks from disk as background threads. The number of threads currently is throttled
by {{ozone.block.deleting.container.limit.per.interval}}, but there is a potential problem.
Containers are sorted so it always fetch same of containers, we need to fix this by creating
an API in {{ContainerManagerImpl}} to get a shuffled list of containers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message