hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-822) Appends to already-finalized blocks can rename across volumes
Date Fri, 15 Jan 2010 22:46:54 GMT

    [ https://issues.apache.org/jira/browse/HDFS-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800973#action_12800973
] 

Todd Lipcon commented on HDFS-822:
----------------------------------

bq. So basically this patch delays the decision to check disk space as opposed to Todd's proposal.

Only problem with this is that it now ignores the dfs.du.reserved setting, since that's only
set at allocation-time. Maybe that's OK, since there are plenty of other ways to overrun the
reserved space - just wanted to bring it up.

> Appends to already-finalized blocks can rename across volumes
> -------------------------------------------------------------
>
>                 Key: HDFS-822
>                 URL: https://issues.apache.org/jira/browse/HDFS-822
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Hairong Kuang
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HDFS-822.patch, HDFS-822.patch
>
>
> This is a performance thing. As I understand the code in FSDataset.append, if the block
is already finalized, it needs to move it into the RBW directory so it can go back into a
"being written" state. This is done using volumes.getNextVolume without preference to the
volume that the block currently exists on. It seems to me that this could cause a lot of slow
cross-volume copies on applications that periodically append/close/append/close a file. Instead,
getNextVolume could provide an alternate form that gives preference to a particular volume,
so the rename stays on the same disk.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message