hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hanisha Koneru (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-4025) QJM: Sychronize past log segments to JNs that missed them
Date Wed, 01 Feb 2017 01:33:51 GMT

     [ https://issues.apache.org/jira/browse/HDFS-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hanisha Koneru updated HDFS-4025:
---------------------------------
    Attachment: HDFS-4025.009.patch

Patch v09 has some changes. 
* Checkstyle corrections
* Minor optimizations
* LastSyncedTxId has been dropped. It's possible that a checkpoint is done during sync interval
and some segments are purged. To ensure correct lastSyncedTxId, we would need the checkpointing
information from the NameNode. We can skip this doing this for now and always sync from the
first transaction id. This improvement can be added later on.

Thank you [~jingzhao] for guiding me through with this.

> QJM: Sychronize past log segments to JNs that missed them
> ---------------------------------------------------------
>
>                 Key: HDFS-4025
>                 URL: https://issues.apache.org/jira/browse/HDFS-4025
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha
>    Affects Versions: QuorumJournalManager (HDFS-3077)
>            Reporter: Todd Lipcon
>            Assignee: Hanisha Koneru
>             Fix For: QuorumJournalManager (HDFS-3077)
>
>         Attachments: HDFS-4025.000.patch, HDFS-4025.001.patch, HDFS-4025.002.patch, HDFS-4025.003.patch,
HDFS-4025.004.patch, HDFS-4025.005.patch, HDFS-4025.006.patch, HDFS-4025.007.patch, HDFS-4025.008.patch,
HDFS-4025.009.patch
>
>
> Currently, if a JournalManager crashes and misses some segment of logs, and then comes
back, it will be re-added as a valid part of the quorum on the next log roll. However, it
will not have a complete history of log segments (i.e any individual JN may have gaps in its
transaction history). This mirrors the behavior of the NameNode when there are multiple local
directories specified.
> However, it would be better if a background thread noticed these gaps and "filled them
in" by grabbing the segments from other JournalNodes. This increases the resilience of the
system when JournalNodes get reformatted or otherwise lose their local disk.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message