hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2718) Optimize OP_ADD in edits loading
Date Thu, 02 Feb 2012 02:57:53 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Konstantin Shvachko updated HDFS-2718:
--------------------------------------

    Attachment: editsLoader-trunk.patch
                editsLoader-0.22.patch

I tried to move updateFiles() into EditLogLoader as Aaron suggested. It is possible, but I
didn't like how it looked. Yes this method is called only in EditLogLoader, but by functionality
it clearly belongs to FSDirectory, as it updates the file in the directory tree, same as addFile()
or delete(). And methods should be assigned to classes based on the functionality rather than
usage. So I left it in FSDirectory.

"Unprotected" means as I understand it that the method does not hold the lock, so the naming
was consistent, but I agree we have too many "unprotected" methods for no apparent reason,
so I renamed it as suggested.

If-s are half that way and half another. I am probably guilty of a lot of them, but don't
see the point of fighting it now.
                
> Optimize OP_ADD in edits loading
> --------------------------------
>
>                 Key: HDFS-2718
>                 URL: https://issues.apache.org/jira/browse/HDFS-2718
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.22.0, 0.24.0, 1.0.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: editsLoader-0.22.patch, editsLoader-0.22.patch, editsLoader-0.22.patch,
editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch, editsLoader-trunk.patch
>
>
> During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently.
It first removes the existing INodeFile from the directory tree, then adds it back as a regular
INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This
slows down edits loading. OP_ADD should be done in one shot and retain previously existing
data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message