hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6871) Improve NameNode performance when creating file
Date Wed, 20 Aug 2014 07:02:29 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103527#comment-14103527

Uma Maheswara Rao G commented on HDFS-6871:

Nice catch Yi!

Thanks a lot for working on fixing. I have one comment.
Just skipping the logsync might have issue with NN crash.
 With overwrite true, existing file deleted and removed the blocks from memory. All this delete
op is under fsnlock as it is in startFIle fsn lock.
Once it release fsnlock, NN might be ready for scheduling the block deletions to DN regarding
to deleted file. Just before logsync if NN schedules block deletions and NN crash without
Then on restart of NN, it will get the deleted file back as it did not persist the edits,
but DN would have deleted the blocks already. Then file metadata exist but blocks will be
missed. This condition is not right for NN state.
Lets delete blocks only after logsync.

> Improve NameNode performance when creating file  
> -------------------------------------------------
>                 Key: HDFS-6871
>                 URL: https://issues.apache.org/jira/browse/HDFS-6871
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode, performance
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>            Priority: Critical
>             Fix For: 2.6.0
>         Attachments: HDFS-6871.001.patch, HDFS-6871.002.patch
> Creating file with overwrite flag will cause NN fall into flush edit logs and block other
requests if the file exists.
> When we create a file with overwrite flag (default is true) in HDFS, NN will remove original
file if it exists. In FSNamesystem#startFileInternal, NN already holds the write lock, it
calls {{deleteInt}} if the file exists, there is logSync in {{deleteInt}}. So in this case,
logSync is under write lock, it will heavily affect the NN performance. 
> We should ignore the force logSync in {{deleteInt}} in this case.

This message was sent by Atlassian JIRA

View raw message