hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6871) Improve NameNode performance when creating file
Date Mon, 25 Aug 2014 21:32:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109776#comment-14109776
] 

Daryn Sharp commented on HDFS-6871:
-----------------------------------

Currently, when the write lock is released the blocks have been purged from various data structures.
 Other operations or subsystems waiting on the lock will not think the blocks exist when they
acquire the lock.  Now the blocks do appear to exist for a brief period after the lock is
released, albeit they are effectively orphaned blocks.  Is this safe?  The answer is probably
yes, but recent 2.x edit log corruption bugs have installed a healthy fear in me.

> Improve NameNode performance when creating file  
> -------------------------------------------------
>
>                 Key: HDFS-6871
>                 URL: https://issues.apache.org/jira/browse/HDFS-6871
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode, performance
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>            Priority: Critical
>             Fix For: 2.6.0
>
>         Attachments: HDFS-6871.001.patch, HDFS-6871.002.patch, HDFS-6871.003.patch
>
>
> Creating file with overwrite flag will cause NN fall into flush edit logs and block other
requests if the file exists.
> When we create a file with overwrite flag (default is true) in HDFS, NN will remove original
file if it exists. In FSNamesystem#startFileInternal, NN already holds the write lock, it
calls {{deleteInt}} if the file exists, there is logSync in {{deleteInt}}. So in this case,
logSync is under write lock, it will heavily affect the NN performance. 
> We should ignore the force logSync in {{deleteInt}} in this case.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message