hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Liang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
Date Thu, 03 Aug 2017 21:08:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113492#comment-16113492

Chen Liang commented on HDFS-12196:

Thanks [~cheersyang] for the reply and the updated patch!

I'm still concerned about the result caching though...I agree with you that it can save RPC
calls, but the thing is it can be very tricky to get the caching work properly...more specifically:

1. One potential issue is that it seems there is no purging of the cache. i.e. once an entry
gets added to the {{resultList}}, it never gets removed. So the size of {{resultList}} will
monolithically increase to the point where no more entries can be added and then the result
cache will no longer be helpful for any further entry. Also since all entries are just being
added when there is space, if there is no more query for those cached result, without purging,
we may end up just holding a bunch of objects in memory that will never be useful. I think
ideally the cache should probably be removing entries based on factors such as access time,
frequency etc. In short, the cache needs to have purging.

2. Seems there is no easy way to check the current {{resultList}} efficiently. Any service
using it will have to check the entire cache. Namely, since it is a list, so even if some
service wants to check if the result of a call is cached in it before making the call, it
still needs to iterate over all the entries in it, so if the list is large enough, it could
be even less efficient than contacting datanodes. In short, the cache needs to have fast lookup.
Another side thing is that the entire {{resultList}} object is exposed and returned to caller,
a bug in caller can easily mess up the list...

I'm not against adding cache here, just saying we should probably pay a little attention.
If we really want cache here, it shouldn't be hard to implement a cache on top of either [LRUMap|https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/map/LRUMap.html]
OR [Guava loading cache|https://github.com/google/guava/wiki/CachesExplained], which should
make purging and fast lookup very easy, even trivial. (One example is the cache in {{XceiverClientManager}},
just FYI).

> Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
> ----------------------------------------------------------------------------------------------
>                 Key: HDFS-12196
>                 URL: https://issues.apache.org/jira/browse/HDFS-12196
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>         Attachments: HDFS-12196-HDFS-7240.001.patch, HDFS-12196-HDFS-7240.002.patch
> Implement a recycling service running on datanode to delete stale blocks.  The recycling
service scans staled blocks for each container and delete chunks and references periodically.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message