hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6038) Allow JournalNode to handle editlog produced by new release with future layoutversion
Date Fri, 14 Mar 2014 00:49:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934370#comment-13934370
] 

Hadoop QA commented on HDFS-6038:
---------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12634559/HDFS-6038.008.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 16 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal:

                  org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

                                      The test build failed in hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal


    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6398//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6398//console

This message is automatically generated.

> Allow JournalNode to handle editlog produced by new release with future layoutversion
> -------------------------------------------------------------------------------------
>
>                 Key: HDFS-6038
>                 URL: https://issues.apache.org/jira/browse/HDFS-6038
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: journal-node, namenode
>            Reporter: Haohui Mai
>            Assignee: Jing Zhao
>         Attachments: HDFS-6038.000.patch, HDFS-6038.001.patch, HDFS-6038.002.patch, HDFS-6038.003.patch,
HDFS-6038.004.patch, HDFS-6038.005.patch, HDFS-6038.006.patch, HDFS-6038.007.patch, HDFS-6038.008.patch,
editsStored
>
>
> In HA setup, the JNs receive edit logs (blob) from the NN and write into edit log files.
In order to write well-formed edit log files, the JNs prepend a header for each edit log file.
The problem is that the JN hard-codes the version (i.e., {{NameNodeLayoutVersion}} in the
edit log, therefore it generates incorrect edit logs when the newer release bumps the {{NameNodeLayoutVersion}}
during rolling upgrade.
> In the meanwhile, currently JN tries to decode the in-progress editlog segment in order
to know the last txid in the segment. In the rolling upgrade scenario, the JN with the old
software may not be able to correctly decode the editlog generated by the new software.
> This jira makes the following changes to allow JN to handle editlog produced by software
with future layoutversion:
> 1. Change the NN--JN startLogSegment RPC signature and let NN specify the layoutversion
for the new editlog segment.
> 2. Persist a length field for each editlog op to indicate the total length of the op.
Instead of calling EditLogFileInputStream#validateEditLog to get the last txid of an in-progress
editlog segment, a new method scanEditLog is added and used by JN which does not decode each
editlog op but uses the length to quickly jump to the next op.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message