hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-11292) log lastWrittenTxId in logSyncAll
Date Wed, 04 Jan 2017 19:03:58 GMT
Yongjun Zhang created HDFS-11292:

             Summary: log lastWrittenTxId in logSyncAll
                 Key: HDFS-11292
                 URL: https://issues.apache.org/jira/browse/HDFS-11292
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: hdfs
            Reporter: Yongjun Zhang

For the issue reported in HDFS-10943, even after HDFS-7964's fix is included, the problem
still exists, this means there might be some synchronization issue.

To diagnose that, create this jira to report the lastWrittenTxId info in {{logSyncAll()}}
call, such that we can compare against the error message reported in HDFS-7964

Specifically, there is two possibility for the HDFS-10943 issue:

1. {{logSyncAll()}} (statement A in the code quoted below) doesn't flush all requested txs
for some reason

2.  {{logSyncAll()}} does flush all requested txs, but some new txs sneaked in between A and
B. It's observed that the lastWrittenTxId in B and C are the same.

This proposed reporting would help confirming if 2 is true.

 public synchronized void endCurrentLogSegment(boolean writeEndTxn) {
    LOG.info("Ending log segment " + curSegmentTxId);
        "Bad state: %s", state);

    if (writeEndTxn) {
    // always sync to ensure all edits are flushed.
A.    logSyncAll();

B.    printStatistics(true);

    final long lastTxId = getLastWrittenTxId();

    try {
C.      journalSet.finalizeLogSegment(curSegmentTxId, lastTxId);
      editLogStream = null;
    } catch (IOException e) {
      //All journals have failed, it will be handled in logSync.

    state = State.BETWEEN_LOG_SEGMENTS;

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message