hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8964) When validating the edit log, do not read at or beyond the file offset that is being written
Date Thu, 03 Sep 2015 22:03:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729882#comment-14729882

Hudson commented on HDFS-8964:

SUCCESS: Integrated in Hadoop-Hdfs-trunk #2269 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2269/])
HDFS-8964. When validating the edit log, do not read at or beyond the file offset that is
being written (Zhe Zhang via Colin P. McCabe) (cmccabe: rev 53c38cc89ab979ec47557dcfa7affbad20578c0a)
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckPointForSecurityTokens.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java

> When validating the edit log, do not read at or beyond the file offset that is being
> --------------------------------------------------------------------------------------------
>                 Key: HDFS-8964
>                 URL: https://issues.apache.org/jira/browse/HDFS-8964
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: journal-node, namenode
>    Affects Versions: 2.7.1
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>             Fix For: 2.8.0
>         Attachments: HDFS-8964.00.patch, HDFS-8964.01.patch, HDFS-8964.02.patch, HDFS-8964.03.patch,
HDFS-8964.04.patch, HDFS-8964.05.patch, HDFS-8964.06.patch
> NN/JN validates in-progress edit log files in multiple scenarios, via {{EditLogFile#validateLog}}.
The method scans through the edit log file to find the last transaction ID.
> However, an in-progress edit log file could be actively written to, which creates a race
condition and causes incorrect data to be read (and later we attempt to interpret the data
as ops).  This causes problems for INotify, which reads edit log entries while the edit log
is still being written.
> Currently {{validateLog}} is used in 3 places:
> # NN {{getEditsFromTxid}}
> # JN {{getEditLogManifest}}
> # NN/JN {{recoverUnfinalizedSegments}}
> In the first two scenarios we should provide a maximum TxId to validate in the in-progress
file. The 3rd scenario won't cause a race condition because only non-current in-progress edit
log files are validated.
> {{validateLog}} is actually only used with in-progress files, and could use a better
name and Javadoc.

This message was sent by Atlassian JIRA

View raw message