Return-Path: X-Original-To: apmail-hadoop-common-dev-archive@www.apache.org Delivered-To: apmail-hadoop-common-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C22B910B4E for ; Fri, 3 Jan 2014 23:17:53 +0000 (UTC) Received: (qmail 84909 invoked by uid 500); 3 Jan 2014 23:17:50 -0000 Delivered-To: apmail-hadoop-common-dev-archive@hadoop.apache.org Received: (qmail 84762 invoked by uid 500); 3 Jan 2014 23:17:50 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 84413 invoked by uid 99); 3 Jan 2014 23:17:50 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jan 2014 23:17:50 +0000 Date: Fri, 3 Jan 2014 23:17:50 +0000 (UTC) From: "Andrei Savu (JIRA)" To: common-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HADOOP-10203) Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Andrei Savu created HADOOP-10203: ------------------------------------ Summary: Connection leak in Jets3tNativeFileSystemStore#retrieveMetadata Key: HADOOP-10203 URL: https://issues.apache.org/jira/browse/HADOOP-10203 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Environment: CDH 2.0.0-cdh4.5.0 (30821ec616ee7a21ee8447949b7c6208a8f1e7d8) Reporter: Andrei Savu Attachments: HADOOP-10203.patch Jets3tNativeFileSystemStore#retrieveMetadata is leaking connections. This affects any client that tries to read many small files very quickly (e.g. distcp from s3 to hdfs with small files blocks due to connection pool starvation). This is not a problem for larger files because when the GC runs any connection that's out of scope will be released in #finalize(). We are seeing the following log messages as a symptom of this problem: {noformat} 13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Attempting to release HttpMethod in finalize() as its response data stream has gone out of scope. This attempt will not always succeed and cannot be relied upon! Please ensure response data streams are always fully consumed or closed to avoid HTTP connection starvation. 13/12/26 13:40:01 WARN httpclient.HttpMethodReleaseInputStream: Successfully released HttpMethod in finalize(). You were lucky this time... Please ensure response data streams are always fully consumed or closed. {noformat} -- This message was sent by Atlassian JIRA (v6.1.5#6160)