hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yiqun Lin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13264) Hadoop HDFS - DFSOutputStream close method fails to clean up resources in case no hdfs datanodes are accessible
Date Thu, 16 Jun 2016 03:20:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332987#comment-15332987
] 

Yiqun Lin commented on HADOOP-13264:
------------------------------------

I think this problem is different with HDFS-9812. HDFS-9812 solved the problem that {{datastreamer}}
thread not closed when failures happened in flushing data, and these logic was done in {{closeImpl}}.
In this probloem, if the method {{closeImpl}} threw the IOException, the dfsClient.endFileLease(fileId)
will not be called. If we want to fix this, I suggest that we would be better  to keep the
synchronized block code, like this:

{code}
  public void close() throws IOException {
    boolean threwException = false;
    synchronized (this) {
      try (TraceScope ignored =
          dfsClient.newPathTraceScope("DFSOutputStream#close", src)) {
        closeImpl();
      } catch (IOException ioe) {
        threwException = true;
      }
    }
    dfsClient.endFileLease(fileId);
    if (threwException) {
      throw new IOException("Exception happened in closing the output stream.");
    }
  }
{code}

Correct me if I am wrong, thanks.

> Hadoop HDFS - DFSOutputStream close method fails to clean up resources in case no hdfs
datanodes are accessible 
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-13264
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13264
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.7.2
>            Reporter: Seb Mo
>
> Using:
> hadoop-hdfs\2.7.2\hadoop-hdfs-2.7.2-sources.jar!\org\apache\hadoop\hdfs\DFSOutputStream.java
> Close method fails when the client can't connect to any data nodes. When re-using the
same DistributedFileSystem in the same JVM, if all the datanodes can't be accessed, then this
causes a memory leak as the DFSClient#filesBeingWritten map is never cleared after that.
> See test program provided by [~sebyonthenet] in comments below.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message