hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8964) When validating the edit log, do not read at or beyond the file offset that is being written
Date Thu, 03 Sep 2015 18:52:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729567#comment-14729567
] 

Zhe Zhang commented on HDFS-8964:
---------------------------------

Thanks Colin for reviewing!

> When validating the edit log, do not read at or beyond the file offset that is being
written
> --------------------------------------------------------------------------------------------
>
>                 Key: HDFS-8964
>                 URL: https://issues.apache.org/jira/browse/HDFS-8964
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: journal-node, namenode
>    Affects Versions: 2.7.1
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>             Fix For: 2.8
>
>         Attachments: HDFS-8964.00.patch, HDFS-8964.01.patch, HDFS-8964.02.patch, HDFS-8964.03.patch,
HDFS-8964.04.patch, HDFS-8964.05.patch, HDFS-8964.06.patch
>
>
> NN/JN validates in-progress edit log files in multiple scenarios, via {{EditLogFile#validateLog}}.
The method scans through the edit log file to find the last transaction ID.
> However, an in-progress edit log file could be actively written to, which creates a race
condition and causes incorrect data to be read (and later we attempt to interpret the data
as ops).  This causes problems for INotify, which reads edit log entries while the edit log
is still being written.
> Currently {{validateLog}} is used in 3 places:
> # NN {{getEditsFromTxid}}
> # JN {{getEditLogManifest}}
> # NN/JN {{recoverUnfinalizedSegments}}
> In the first two scenarios we should provide a maximum TxId to validate in the in-progress
file. The 3rd scenario won't cause a race condition because only non-current in-progress edit
log files are validated.
> {{validateLog}} is actually only used with in-progress files, and could use a better
name and Javadoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message