hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8531) Append failed due to unreleased lease from previous appender with quota exceeded exception
Date Thu, 04 Jun 2015 04:51:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572137#comment-14572137
] 

Xiaoyu Yao commented on HDFS-8531:
----------------------------------

In this case, the 1st appender still get the lease because the first partial block passed
FSNamesystem#verifyQuotaForUCBlock(). It is the subsequent blocks that exceed the space quota.
When 2nd appender tries to recover the lease with FSNamesystem#recoverLeaseInternal(), it
does not handle this where the last block is in COMPLETE state which we should allow to recover
lease immediately. 

{code}

 if (lastBlock != null
              && lastBlock.getBlockUCState() == BlockUCState.UNDER_RECOVERY) {
            throw new RecoveryInProgressException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "another recovery is in progress by "
                        + clientName + " on " + uc.getClientMachine()));
          } else {
            throw new AlreadyBeingCreatedException(
                op.getExceptionMessage(src, holder, clientMachine,
                    "this file lease is currently owned by "
                        + clientName + " on " + uc.getClientMachine()));
          }
{code}

> Append failed due to unreleased lease from previous appender with quota exceeded exception
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-8531
>                 URL: https://issues.apache.org/jira/browse/HDFS-8531
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Xiaoyu Yao
>            Assignee: Xiaoyu Yao
>
> Append operation fails if
> 1. Set SpaceQuota to 3G for /user/hrt_qa/heterogenous
> 2. Copy 1GB file to /user/hrt_qa/heterogenous
> 3. Run 'hdfs dfs -appendToFile' to append 1GB file to the already copied 1GB file in
/user/hrt_qa/heterogenous
> 4. Append fails with Quota exceed message
> 5. Increase the Quota to 5G for /user/hrt_qa/heterogenous
> 6. Run the Append message again and the following error message comes.
> {code}
> appendToFile: Failed to APPEND_FILE /user/hrt_qa/quotaPerHerterogenousStorage/1GBFile_1/part-m-00000
for DFSClient_NONMAPREDUCE_1431576997_1 on 172.31.37.190 because this file lease is currently
owned by DFSClient_NONMAPREDUCE_-231994503_1 on 172.31.37.190
> {code}
> 7. Wait for a while (lease soft limit -- 1 min) and retry append it succeeds after the
previous lease expire. 
> This seems to relate to HDFS-7587
> "
> When a client was trying to append to the file, the remaining space quota was very small.
This caused a failure in prepareFileForWrite(), but after the inode was already converted
for writing and a lease added. Since these were not undone when the quota violation was detected,
the file was left in under-construction with an active lease without edit logging OP_ADD.
> A subsequent append() eventually caused a lease recovery after the soft limit period.
This resulted in commitBlockSynchronization(), which closed the file with OP_CLOSE being logged."
> The fix in HDFS-7587 addressed the edit log corruption issue but does not handle the
problem within soft limit period where other clients can't append.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message