hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrei Savu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
Date Fri, 03 Jan 2014 23:17:50 GMT
Andrei Savu created HADOOP-10203:
------------------------------------

             Summary: Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata 
                 Key: HADOOP-10203
                 URL: https://issues.apache.org/jira/browse/HADOOP-10203
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
         Environment: CDH 2.0.0-cdh4.5.0 (30821ec616ee7a21ee8447949b7c6208a8f1e7d8) 
            Reporter: Andrei Savu
         Attachments: HADOOP-10203.patch

Jets3tNativeFileSystemStore#retrieveMetadata  is leaking connections. 

This affects any client that tries to read many small files very quickly (e.g. distcp from
s3 to hdfs with small files blocks due to connection pool starvation). 

This is not a problem for larger files because when the GC runs any connection that's out
of scope will be released in #finalize().

We are seeing the following log messages as a symptom of this problem:

{noformat}
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to release HttpMethod
in finalize() as its response data stream has gone out of scope. This attempt will not always
succeed and cannot be relied upon! Please ensure response data streams are always fully consumed
or closed to avoid HTTP connection starvation.
13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully released HttpMethod
in finalize(). You were lucky this time... Please ensure response data streams are always
fully consumed or closed.
{noformat}




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message