hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Zhang (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-14758) Decrease lease hard limit
Date Sat, 24 Aug 2019 02:28:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914770#comment-16914770
] 

Chen Zhang edited comment on HDFS-14758 at 8/24/19 2:27 AM:
------------------------------------------------------------

Hi [~jojochuang] and [~hemanthboyina], HDFS-14694 actually want to address the broken state
of open file for long running client(e.g. HBase), in this situation hard limit can't help
to recover file state.
Decreasing hard limit may be helpful in the scenarios that client crash or client dead(closed)
on any exception (e.g. MR job), but in our experience, if some client want to access the file
before hard limit and found that the file is not closed correctly, it can call recoverLease()
and try again, so I think it's not necessary to decrease hard limit.

I mean, even we decreased the hard limit to 1 hour or 30min, it's still possible for some
client to access the file before hard limit exceeded. And calling recoverLease() is the right
solution in this situation.


was (Author: zhangchen):
Hi [~jojochuang] and [~hemanthboyina], HDFS-14694 actually want to address the broken state
of open file for long running client(e.g. HBase), in this situation hard limit can't help
to recover file state.
Decreasing hard limit may be helpful in the scenarios that client crash or client dead(closed)
on any exception (e.g. MR job), but in our experience, if some client want to access the file
before hard limit and found that the file is not closed correctly, it can call recoverLease()
and try again, so I think it's not necessary to decrease hard limit.
I mean, even we decreased the hard limit to 1 hour or 30min, it's still possible for some
client to access the file before hard limit exceeded. And calling recoverLease() is the right
solution in this situation.

> Decrease lease hard limit
> -------------------------
>
>                 Key: HDFS-14758
>                 URL: https://issues.apache.org/jira/browse/HDFS-14758
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Eric Payne
>            Assignee: hemanthboyina
>            Priority: Minor
>
> The hard limit is currently hard-coded to be 1 hour. This also determines the NN automatic
lease recovery interval. Something like 20 min will make more sense.
> After the 5 min soft limit, other clients can recover the lease. If no one else takes
the lease away, the original client still can renew the lease within the hard limit. So even
after a NN full GC of 8 minutes, leases can be still valid.
> However, there is one risk in reducing the hard limit. E.g. Reduced to 20 min. If the
NN crashes and the manual failover takes more than 20 minutes, clients will abort.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message