hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata
Date Sat, 04 Jan 2014 12:26:53 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13862284#comment-13862284
] 

Steve Loughran commented on HADOOP-10203:
-----------------------------------------

why the move from the S3 classes to the Storage ones? I don't see it compiling without imports
and some other changes -but I'll wait to see what Jenkins says there

> Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata 
> ----------------------------------------------------------------
>
>                 Key: HADOOP-10203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10203
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>         Environment: CDH 2.0.0-cdh4.5.0 (30821ec616ee7a21ee8447949b7c6208a8f1e7d8) 
>            Reporter: Andrei Savu
>            Assignee: Andrei Savu
>         Attachments: HADOOP-10203-trunk.patch, HADOOP-10203.patch
>
>
> Jets3tNativeFileSystemStore#retrieveMetadata  is leaking connections. 
> This affects any client that tries to read many small files very quickly (e.g. distcp
from s3 to hdfs with small files blocks due to connection pool starvation). 
> This is not a problem for larger files because when the GC runs any connection that's
out of scope will be released in #finalize().
> We are seeing the following log messages as a symptom of this problem:
> {noformat}
> 13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to release
HttpMethod in finalize() as its response data stream has gone out of scope. This attempt will
not always succeed and cannot be relied upon! Please ensure response data streams are always
fully consumed or closed to avoid HTTP connection starvation.
> 13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully released
HttpMethod in finalize(). You were lucky this time... Please ensure response data streams
are always fully consumed or closed.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message