hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8531) Append failed due to unreleased lease from previous appender with quota exceeded exception
Date Thu, 04 Jun 2015 04:51:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572137#comment-14572137

Xiaoyu Yao commented on HDFS-8531:

In this case, the 1st appender still get the lease because the first partial block passed
FSNamesystem#verifyQuotaForUCBlock(). It is the subsequent blocks that exceed the space quota.
When 2nd appender tries to recover the lease with FSNamesystem#recoverLeaseInternal(), it
does not handle this where the last block is in COMPLETE state which we should allow to recover
lease immediately. 


 if (lastBlock != null
              && lastBlock.getBlockUCState() == BlockUCState.UNDER_RECOVERY) {
            throw new RecoveryInProgressException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "another recovery is in progress by "
                        + clientName + " on " + uc.getClientMachine()));
          } else {
            throw new AlreadyBeingCreatedException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "this file lease is currently owned by "
                        + clientName + " on " + uc.getClientMachine()));

> Append failed due to unreleased lease from previous appender with quota exceeded exception
> ------------------------------------------------------------------------------------------
>                 Key: HDFS-8531
>                 URL: https://issues.apache.org/jira/browse/HDFS-8531
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Xiaoyu Yao
>            Assignee: Xiaoyu Yao
> Append operation fails if
> 1. Set SpaceQuota to 3G for /user/hrt_qa/heterogenous
> 2. Copy 1GB file to /user/hrt_qa/heterogenous
> 3. Run 'hdfs dfs -appendToFile' to append 1GB file to the already copied 1GB file in
> 4. Append fails with Quota exceed message
> 5. Increase the Quota to 5G for /user/hrt_qa/heterogenous
> 6. Run the Append message again and the following error message comes.
> {code}
> appendToFile: Failed to APPEND_FILE /user/hrt_qa/quotaPerHerterogenousStorage/1GBFile_1/part-m-00000
for DFSClient_NONMAPREDUCE_1431576997_1 on because this file lease is currently
owned by DFSClient_NONMAPREDUCE_-231994503_1 on
> {code}
> 7. Wait for a while (lease soft limit -- 1 min) and retry append it succeeds after the
previous lease expire. 
> This seems to relate to HDFS-7587
> "
> When a client was trying to append to the file, the remaining space quota was very small.
This caused a failure in prepareFileForWrite(), but after the inode was already converted
for writing and a lease added. Since these were not undone when the quota violation was detected,
the file was left in under-construction with an active lease without edit logging OP_ADD.
> A subsequent append() eventually caused a lease recovery after the soft limit period.
This resulted in commitBlockSynchronization(), which closed the file with OP_CLOSE being logged."
> The fix in HDFS-7587 addressed the edit log corruption issue but does not handle the
problem within soft limit period where other clients can't append.

This message was sent by Atlassian JIRA

View raw message