hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-11292) log lastWrittenTxId etc info in logSyncAll
Date Thu, 05 Jan 2017 03:20:58 GMT

     [ https://issues.apache.org/jira/browse/HDFS-11292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yongjun Zhang updated HDFS-11292:
---------------------------------
    Summary: log lastWrittenTxId etc info in logSyncAll  (was: log lastWrittenTxId in logSyncAll)

> log lastWrittenTxId etc info in logSyncAll
> ------------------------------------------
>
>                 Key: HDFS-11292
>                 URL: https://issues.apache.org/jira/browse/HDFS-11292
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>         Attachments: HDFS-11292.001.patch, HDFS-11292.002.patch
>
>
> For the issue reported in HDFS-10943, even after HDFS-7964's fix is included, the problem
still exists, this means there might be some synchronization issue.
> To diagnose that, create this jira to report the lastWrittenTxId info in {{logSyncAll()}}
call, such that we can compare against the error message reported in HDFS-7964
> Specifically, there is two possibility for the HDFS-10943 issue:
> 1. {{logSyncAll()}} (statement A in the code quoted below) doesn't flush all requested
txs for some reason
> 2.  {{logSyncAll()}} does flush all requested txs, but some new txs sneaked in between
A and B. It's observed that the lastWrittenTxId in B and C are the same.
> This proposed reporting would help confirming if 2 is true.
> {code}
>  public synchronized void endCurrentLogSegment(boolean writeEndTxn) {
>     LOG.info("Ending log segment " + curSegmentTxId);
>     Preconditions.checkState(isSegmentOpen(),
>         "Bad state: %s", state);
>     if (writeEndTxn) {
>       logEdit(LogSegmentOp.getInstance(cache.get(),
>           FSEditLogOpCodes.OP_END_LOG_SEGMENT));
>     }
>     // always sync to ensure all edits are flushed.
> A.    logSyncAll();
> B.    printStatistics(true);
>     final long lastTxId = getLastWrittenTxId();
>     try {
> C.      journalSet.finalizeLogSegment(curSegmentTxId, lastTxId);
>       editLogStream = null;
>     } catch (IOException e) {
>       //All journals have failed, it will be handled in logSync.
>     }
>     state = State.BETWEEN_LOG_SEGMENTS;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message