hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-6315) Decouple recording edit logs from FSDirectory
Date Fri, 06 Jun 2014 18:40:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14020181#comment-14020181
] 

Haohui Mai edited comment on HDFS-6315 at 6/6/14 6:38 PM:
----------------------------------------------------------

The v2 patch moves all edit log operations into {{FSNameSystem}} except for {{rename}} and
{{delete}}. These two functions require more changes thus I plan to address them in a separate
jira.

The patch makes the following changes:

# It moves the edit log operations of {{setReplication}}, {{setPermission}}, {{setOwner}},
{{concat}}, {{setQuota}}, and all ACL / XAttr operations into their respective callers.
# The patch moves {{persistBlock}} and {{persistNewBlock}} into FSDirectory. The original
code protects these two methods with the lock of the {{FSDirectory}}. The patch moves them
out of the lock because these two methods only log the edit logs, where the logging itself
has been made synchronized in the {{EditLog}} class.

With the above changes, this patch no longer leaks the lock of the {{FSDirectory}} into {{FSNameSystem}}.
[~daryn], do you have any more comments?


was (Author: wheat9):
The v2 patch moves all edit log operations into {{FSNameSystem}} except for {{rename}} and
{{delete}}. These two functions require more changes thus I plan to address them in a separate
jira.

The patch makes the following changes:

# It moves the edit log operations of {{setReplication}}, {{setPermission}}, {{setOwner}},
{{concat}}, {{setQuota}}, and all ACL / XAttr operations into their respective callees.
# The patch moves {{persistBlock}} and {{persistNewBlock}} into FSDirectory. The original
code protects these two methods with the lock of the {{FSDirectory}}. The patch moves them
out of the lock because these two methods only log the edit logs, where the logging itself
has been made synchronized in the {{EditLog}} class.

With the above changes, this patch no longer leaks the lock of the {{FSDirectory}} into {{FSNameSystem}}.
[~daryn], do you have any more comments?

> Decouple recording edit logs from FSDirectory
> ---------------------------------------------
>
>                 Key: HDFS-6315
>                 URL: https://issues.apache.org/jira/browse/HDFS-6315
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>         Attachments: HDFS-6315.000.patch, HDFS-6315.001.patch, HDFS-6315.002.patch
>
>
> Currently both FSNamesystem and FSDirectory record edit logs. This design requires both
FSNamesystem and FSDirectory to be tightly coupled together to implement a durable namespace.
> This jira proposes to separate the responsibility of implementing the namespace and providing
durability with edit logs. Specifically, FSDirectory implements the namespace (which should
have no edit log operations), and FSNamesystem implement durability by recording the edit
logs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message