hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8531) Append failed due to unreleased lease from previous appender with quota exceeded exception
Date Thu, 04 Jun 2015 22:13:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573683#comment-14573683

Xiaoyu Yao commented on HDFS-8531:

After compare the pros and cons of the existing lease recovery logic and the proposed one,
I found the existing one actually is actually better. 

The existing approach: new appender wait (soft limit) and retry OR explicitly force reclaim
other appender's lease.
Pros: appender has better control and less thrashing
Cons: appender will have to wait or force reclaim lease from others.

The proposed: allowing one appender to implicitly reclaim other appender's lease as long as
the last block is COMPLETE.
Pros: Guarantee fairness among appenders
Cons: Very aggressive, may causes unnecessary thrashing among appenders.

> Append failed due to unreleased lease from previous appender with quota exceeded exception
> ------------------------------------------------------------------------------------------
>                 Key: HDFS-8531
>                 URL: https://issues.apache.org/jira/browse/HDFS-8531
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.7.0
>            Reporter: Sumana Sathish
>            Assignee: Xiaoyu Yao
>         Attachments: HDFS-8531.00.patch
> Append operation fails if
> 1. Set SpaceQuota to 3G for /user/hrt_qa/heterogenous
> 2. Copy 1GB file to /user/hrt_qa/heterogenous
> 3. Run 'hdfs dfs -appendToFile' to append 1GB file to the already copied 1GB file in
> 4. Append fails with Quota exceed message
> 5. Increase the Quota to 5G for /user/hrt_qa/heterogenous
> 6. Run the Append message again and the following error message comes.
> {code}
> appendToFile: Failed to APPEND_FILE /user/hrt_qa/quotaPerHerterogenousStorage/1GBFile_1/part-m-00000
for DFSClient_NONMAPREDUCE_1431576997_1 on because this file lease is currently
owned by DFSClient_NONMAPREDUCE_-231994503_1 on
> {code}
> 7. Wait for a while (lease soft limit -- 1 min) and retry append it succeeds after the
previous lease expire. 
> This seems to relate to HDFS-7587
> "
> When a client was trying to append to the file, the remaining space quota was very small.
This caused a failure in prepareFileForWrite(), but after the inode was already converted
for writing and a lease added. Since these were not undone when the quota violation was detected,
the file was left in under-construction with an active lease without edit logging OP_ADD.
> A subsequent append() eventually caused a lease recovery after the soft limit period.
This resulted in commitBlockSynchronization(), which closed the file with OP_CLOSE being logged."
> The fix in HDFS-7587 addressed the edit log corruption issue but does not handle the
problem within soft limit period where other clients can't append.

This message was sent by Atlassian JIRA

View raw message