hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-12142) Files may be closed before streamer is done
Date Fri, 14 Jul 2017 15:41:00 GMT
Daryn Sharp created HDFS-12142:

             Summary: Files may be closed before streamer is done
                 Key: HDFS-12142
                 URL: https://issues.apache.org/jira/browse/HDFS-12142
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: hdfs
    Affects Versions: 2.8.0
            Reporter: Daryn Sharp

We're encountering multiple cases of clients calling updateBlockForPipeline on completed blocks.
 Initial analysis is the client closes a file, completeFile succeeds, then it immediately
attempts recovery.  The exception is swallowed on the client, only logged on the NN by checkUCBlock.

The problem "appears" to be benign (no data loss) but it's unproven if the issue always occurs
for successfully closed files.  There appears to be very poor coordination between the dfs
output stream's threads which leads to races that confuse the streamer thread – which probably
should have been joined before returning from close.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message