hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Liang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12626) Ozone : delete open key entries that will no longer be closed
Date Tue, 10 Oct 2017 20:27:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199303#comment-16199303

Chen Liang commented on HDFS-12626:

There are definitely different ways to handle this, but the simpler solution I'm thinking
is that, simply having a thread that periodically checks all the open key entries, if they
have been there for a very long time, say, more than X hours, we treat them as dead entries
and remove them. The tricky thing is what this X should be. Because if X is too small, then
the client might still be writing after X hours. Basically, X should be longer than the time
any single key write would take. Here I'm thinking of something like X = 24 hours. Because
I don't see a use case where a single key write would take this long (maybe I'm wrong). Also,
since I assume client crash is relatively rare so the number of dead entries shouldn't be
too large, so it should be okay to reclaim them only once every day.

Any thoughts? [~xyao] [~anu]

> Ozone : delete open key entries that will no longer be closed
> -------------------------------------------------------------
>                 Key: HDFS-12626
>                 URL: https://issues.apache.org/jira/browse/HDFS-12626
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Chen Liang
>            Assignee: Chen Liang
> HDFS-12543 introduced the notion of "open key" where when a key is opened, an open key
entry gets persisted, only after client calls a close will this entry be made visible. One
issue is that if the client does not call close (e.g. failed), then that open key entry will
never be deleted from meta data. This JIRA tracks this issue.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message