hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8480) Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs instead of copying them
Date Wed, 29 Jul 2015 03:26:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645399#comment-14645399
] 

Ming Ma commented on HDFS-8480:
-------------------------------

Thanks [~zhz].

The test code simulates older version edit by creating EditLogFileOutputStream with the prior
version. However, the FSEditLogOp's writeFields implementation doesn't distinguish different
versions and always write with the latest features.

For example, if this patch is applied directly on the top of the 2.6 release where CURRENT_LAYOUT_VERSION
is set to -60, the test will fail because even though EditLogFileOutputStream is created with
version -59, OP_ADD's writeFields will add storagePolicyId to the edit. During the edit loading,
it will skip the reading of storagePolicyId given it has version -59 and thus cause checksum
error.

To simulates old edit for test purpose, it seems we need to have FSEditLogOp support writing
optional data depending on the version requested. So in this case, if the version is set to
-59, it shouldn't write storagePolicyId. This functionality seems only useful for testing
upgrade scenario.

Thought?

> Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit
logs instead of copying them
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-8480
>                 URL: https://issues.apache.org/jira/browse/HDFS-8480
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.0
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>            Priority: Critical
>             Fix For: 2.7.1
>
>         Attachments: HDFS-8480.00.patch, HDFS-8480.01.patch, HDFS-8480.02.patch, HDFS-8480.03.patch
>
>
> HDFS-7929 copies existing edit logs to the storage directory of the upgraded {{NameNode}}.
This slows down the upgrade process. This JIRA aims to use hard-linking instead of per-op
copying to achieve the same goal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message