hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Krogen (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (HDFS-14048) DFSOutputStream close() throws exception on subsequent call after DataNode restart
Date Tue, 06 Nov 2018 20:05:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Erik Krogen reopened HDFS-14048:

Re-opening for branch-2 commit.

Sorry for the trouble [~elgoiri], I have just attached the branch-2 patch. Since I'm not sure
if Jenkins will run properly given the branch-2 build issues, I also executed all of the following
tests locally without any failures: {{TestClientProtocolForPipelineRecovery,TestDFSOutputStream,TestClientBlockVerification,TestDatanodeRestart}}

> DFSOutputStream close() throws exception on subsequent call after DataNode restart
> ----------------------------------------------------------------------------------
>                 Key: HDFS-14048
>                 URL: https://issues.apache.org/jira/browse/HDFS-14048
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.3.0
>            Reporter: Erik Krogen
>            Assignee: Erik Krogen
>            Priority: Major
>             Fix For: 3.0.4, 3.1.2, 3.3.0, 3.2.1
>         Attachments: HDFS-14048.000.patch
> We recently discovered an issue in which, during a rolling upgrade, some jobs were failing
with exceptions like (sadly this is the whole stack trace):
> {code}
> java.io.IOException: A datanode is restarting: DatanodeInfoWithStorage[,BP-XXXX,DISK]
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:877)
> {code}
> with an earlier statement in the log like:
> {code}
> INFO [main] org.apache.hadoop.hdfs.DFSClient: A datanode is restarting: DatanodeInfoWithStorage[,BP-XXXX,DISK]
> {code}
> Strangely we did not see any other logs about the {{DFSOutputStream}} failing after waiting
for the DataNode restart. We eventually realized that in some cases {{DFSOutputStream#close()}}
may be called more than once, and that if so, the {{IOException}} above is thrown on the _second_
call to {{close()}} (this is even with HDFS-5335; prior to this it would have been thrown
on all calls to {{close()}} besides the first).
> The problem is that in {{DataStreamer#createBlockOutputStream()}}, after the new output
stream is created, it resets the error states:
> {code}
>         errorState.resetInternalError();
>         // remove all restarting nodes from failed nodes list
>         failed.removeAll(restartingNodes);
>         restartingNodes.clear(); 
> {code}
> But it forgets to clear {{lastException}}. When {{DFSOutputStream#closeImpl()}} is called
a second time, this block is triggered:
> {code}
>     if (isClosed()) {
>       LOG.debug("Closing an already closed stream. [Stream:{}, streamer:{}]",
>           closed, getStreamer().streamerClosed());
>       try {
>         getStreamer().getLastException().check(true);
> {code}
> The second time, {{isClosed()}} is true, so the exception checking occurs and the "Datanode
is restarting" exception is thrown even though the stream has already been successfully closed.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message