Return-Path: Delivered-To: apmail-ws-axis-dev-archive@www.apache.org Received: (qmail 89976 invoked from network); 27 Feb 2009 09:11:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 27 Feb 2009 09:11:05 -0000 Received: (qmail 48003 invoked by uid 500); 27 Feb 2009 09:10:58 -0000 Delivered-To: apmail-ws-axis-dev-archive@ws.apache.org Received: (qmail 47941 invoked by uid 500); 27 Feb 2009 09:10:58 -0000 Mailing-List: contact axis-dev-help@ws.apache.org; run by ezmlm Precedence: bulk Reply-To: axis-dev@ws.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list axis-dev@ws.apache.org Received: (qmail 47932 invoked by uid 99); 27 Feb 2009 09:10:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 01:10:58 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of kdobrik.axis2@googlemail.com designates 209.85.220.168 as permitted sender) Received: from [209.85.220.168] (HELO mail-fx0-f168.google.com) (209.85.220.168) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 09:10:49 +0000 Received: by fxm12 with SMTP id 12so912686fxm.16 for ; Fri, 27 Feb 2009 01:10:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=MA2x7FYTIUPImgLMWezwnDUWmAPcDXmCKfNWG7lXvEc=; b=KSOh1edXUZBmTEI1fPAsdBmv0OEAu6GTZS+6B7k2uwhaTwxqj5P7BVvZ7/bXEzpHqj 02efwmrLR7nNanXS+9AZfazp9W5QG+HubK/ES4nMaLmLP2uowJSLMqqRIv6NWbnmzWqW rFPhBQVLXXwuopXszQaYPOKDbm9kMMIQZJkHs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=j31EkbInIZaX22LpKnqLAWrzu+hjExg5ThHgF7NhhqezZKC7Gqi9WrQ7cPsN4AHFbE dZcImW1MtqSX3K/KB1rmAm0rEp6IAQrNK+rcooURc4/kzP53tHbWFybHBAJzvhva64bp Io8UBBu92mFny9hkz91+IsYs3cK1DvM3wF63E= MIME-Version: 1.0 Received: by 10.181.13.19 with SMTP id q19mr799527bki.53.1235725828281; Fri, 27 Feb 2009 01:10:28 -0800 (PST) In-Reply-To: References: Date: Fri, 27 Feb 2009 11:10:28 +0200 Message-ID: <3b3bfaf80902270110v453110bal126c5d4930267797@mail.gmail.com> Subject: Re: HTTP connection leak and other related issues From: Dobri Kitipov To: axis-dev@ws.apache.org Content-Type: multipart/alternative; boundary=001636c5aace08d4ff0463e2d782 X-Virus-Checked: Checked by ClamAV on apache.org --001636c5aace08d4ff0463e2d782 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi all, I have observed absolutely the same thing these days. I need some more time to analyze the whole picture, but here is my current synthesis of the issue. It seems that http connection release is tightly coupled with the Axis2 builder used to handle and process the response body and the corresponding input streams used. The builder and the InputStream used are based on the HTTP headers fields like "Content-Type" and "Transfer-Encoding" (e.g. Content-Type: text/xml; charset=UTF-8 Transfer-Encoding: chunked). So if we have Content-Type: text/xml; then SOAPBuilder class will be used. If we have type="application/xop+xml" then MIMEBuilder will be used. The successfull story when we have MIMIBuilder: When MIMEBuilder is used then the response Buffered InputStream (IS) is wrrapped (I will use "->" sign as substitution for wrrapped) ChunkedIS -> AutoCloseIS (this has a responseConsumed() method notified when IS.read() returns -1, which means that the response IS has been completely read) -> PushBackIS ->BounderyPushBackIS. The BounderyPushBackIS reads the response stream (see readFromStream(....)) in a cycle till it reaches its end. At every iteration of this cycle a AutoCloseIS checkClose(l) is invoked. So when the end is reached (-1 is returned) then this check causes the invokation of the AutoCloseIS checkClose(...) method. This method invokes notifyWatcher() that in turn invokes responseBodyConsumed() method of the HttpMethodBase class. This causes the release of the http connection which is returned back to the connection pool. So here we have no problem with connection reuse. The bad story we have with SOAPBuilder: When SOAPBuilder is used then the response Buffered InputStream is wrrapped in a ChunkedIS -> AutoCloseIS -> PushBackIS. May be you has noticed that BounderyPushBackIS is not used. As a result the respose IS is not completely read (in fact this is not really correct, it could be read but the invokation of the PushBackIS unread(...) method causes the AutoCloseIS checkClose() method never to return -1). As a result the http connection is not released. And since there is a limit to have 2 connection per host then after the second invokation of the WS client the thread hangs waiting for a free connection. Please, provide us with your comments on this issue. Thank you in advance. Regards, Dobri On Fri, Feb 27, 2009 at 3:54 AM, Alexis Midon wrote: > no taker for an easy patch? > > Alexis > > > On Wed, Feb 25, 2009 at 6:45 PM, Alexis Midon wrote: > >> >> Hi everyone, >> >> All the issues relatives to AXIS2-935 are really messy, some of them are >> closed but their clones are not. Some are flagged as fixed but are obviously >> not. All these issues are really old, so I'd like to take a chance to bring >> them back to your attention, especially before releasing 1.5. >> >> I'll post a description of the issue in this email as a summary all the >> jiras. >> >> By default, ServiceClient uses one HttpConnectionManager per invocation >> [2]. This connection manager will create and provide one connection to >> HTTPSender. The first issue is that by default this connection is never >> released to the pool [3]. if you do zillions of invocations, this leak will >> max out your number of file descriptors. >> >> Your investigations in Axis2 options quickly lead you to the >> REUSE_HTTP_CLIENT option. But this first issue has some unfortunate >> consequences if you activate it. Actually if you do so, a single connection >> manager is shared across all invocations. But because connections are not >> release, the pool is starved after two invocations, and the third invocation >> hangs out indefinitely. :( >> >> If you keep digging you will find the AUTO_RELEASE_CONNECTION option. Its >> sounds like a good lead! Let's try it. If you activate this option the >> connection is properly released -Yahoooo! the leak is fixed - but >> unfortunately a new issue shows up (issue #2, aka AXIS2-3478). >> AbstractHTTPSender passes the stream of the connection to the message >> context [4] , but that the connection is now properly released, so this >> stream is closed before the SOAPBuilder gets a chance to read the response >> body. >> Boom! "IOException: Attempted read on closed stream" >> >> These issues are easily reproducible in versions 1.3, 1.4, 1.5. >> >> I submitted and documented a fix in AXIS2-2931 [5], if you had a chance to >> look at it that would be much appreciate. >> >> >> Alexis >> >> >> [1] >> https://issues.apache.org/jira/browse/AXIS2-935?focusedCommentId=12513543#action_12513543 >> [2] see method getHttpClient in >> https://svn.apache.org/repos/asf/webservices/commons/trunk/modules/transport/modules/http/src/org/apache/axis2/transport/http/AbstractHTTPSender.java >> [3] see method cleanup in >> https://svn.apache.org/repos/asf/webservices/commons/trunk/modules/transport/modules/http/src/org/apache/axis2/transport/http/HTTPSender.java >> [4] see method processResponse in AbstractHTTPSender.java >> [5] >> https://issues.apache.org/jira/browse/AXIS2-2931?focusedCommentId=12676837#action_12676837 >> > > --001636c5aace08d4ff0463e2d782 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi all,
I have observed absolutely the same thing these days. I need som= e more time to analyze the whole picture, but here is my current synthesis = of the issue.

It seems that http connection release is tightly coupl= ed=A0 with the Axis2 builder used to handle and process the response body a= nd the corresponding input streams used. The builder and the InputStream us= ed are based on the HTTP headers fields like "Content-Type" and &= quot;Transfer-Encoding" (e.g. Content-Type: text/xml; charset=3DUTF-8= =A0 Transfer-Encoding: chunked). So if we have Content-Type: text/xml; then= SOAPBuilder class will be used. If we have=A0 type=3D"application/xop= +xml" then MIMEBuilder will be used.

The successfull story when we have MIMIBuilder:

When MIMEBuilder= is used then the response Buffered InputStream (IS) is wrrapped (I will us= e "->" sign as substitution for wrrapped) ChunkedIS -> Auto= CloseIS (this has a responseConsumed() method notified when IS.read() retur= ns -1, which means that the response IS has been completely read) -> Pus= hBackIS ->BounderyPushBackIS. The BounderyPushBackIS reads the response = stream (see readFromStream(....)) in a cycle till it reaches its end. At ev= ery iteration of this cycle a AutoCloseIS checkClose(l) is invoked. So when= the end is reached (-1 is returned) then this check causes the invokation = of the AutoCloseIS checkClose(...)=A0 method. This method invokes notifyWat= cher() that in turn invokes responseBodyConsumed() method of the HttpMethod= Base class. This causes the release of the http connection which is returne= d back to the connection pool. So here we have no problem with connection r= euse.

The bad story we have with SOAPBuilder:

When SOAPBuilder is used= then the response Buffered InputStream is wrrapped in a ChunkedIS -> AutoCloseIS -> PushBackIS. May be you has noticed that BounderyPushBackI= S is not used. As a result the respose IS is not completely read (in fact t= his is not really correct, it could be read but the invokation of the PushB= ackIS unread(...) method causes the AutoCloseIS checkClose() method never t= o return -1). As a result the http connection is not released. And since th= ere is a limit to have 2 connection per host
then after the second invokation of the WS client the thread hangs waiting = for a free connection.

Please, provide us with your comments on this= issue.

Thank you in advance.

Regards,
Dobri

On Fri, Feb 27, 2009 at 3:54 AM, Alexis Midon <midon@intalio.com> wrote:
no taker for an easy patch?

Alexis


On Wed, Feb 25, 2009 at 6:45 PM, Alexis = Midon <midon@intalio.com> wrote:

Hi everyone,

All the issues relatives to AXIS2-935 are really me= ssy, some of them are closed but their clones are not. Some are flagged as = fixed but are obviously not. All these issues are really old, so I'd li= ke to take a chance to bring them back to your attention, especially before= releasing 1.5.

I'll post a description of the issue in this email as a summary all= the jiras.

By default, ServiceClient uses one HttpConnectionManage= r per invocation [2]. This connection manager will create and provide one c= onnection to HTTPSender. The first issue is that by default this connection= is never released to the pool [3]. if you do zillions of invocations, this= leak will max out your number of file descriptors.

Your investigations in Axis2 options quickly lead you to the REUSE_HTTP= _CLIENT option. But this first issue has some unfortunate consequences if y= ou activate it. Actually if you do so, a single connection manager is share= d across all invocations. But because connections are not release, the pool= is starved after two invocations, and the third invocation hangs out indef= initely. :(

If you keep digging you will find the AUTO_RELEASE_CONNECTION option. I= ts sounds like a good lead! Let's try it. If you activate this option t= he connection is properly released -Yahoooo! the leak is fixed - but unfort= unately a new issue shows up (issue #2, aka AXIS2-3478). AbstractHTTPSender= passes the stream of the connection to the message context [4] , but that = the connection is now properly released, so this stream is closed before th= e SOAPBuilder gets a chance to read the response body.
Boom! "IOException: Attempted read on closed stream"

These= issues are easily reproducible in versions 1.3, 1.4, 1.5.

I submitt= ed and documented a fix in AXIS2-2931 [5], if you had a chance to look at i= t that would be much appreciate.


Alexis


[1] https://issues.apache.org/jira/browse/AXIS2-935?focusedCommentId=3D125135= 43#action_12513543
[2] see method getHttpClient in https://svn.a= pache.org/repos/asf/webservices/commons/trunk/modules/transport/modules/htt= p/src/org/apache/axis2/transport/http/AbstractHTTPSender.java
[3] see method cleanup in https://svn.apache.org/repo= s/asf/webservices/commons/trunk/modules/transport/modules/http/src/org/apac= he/axis2/transport/http/HTTPSender.java
[4] see method processResponse in AbstractHTTPSender.java
[5] https://issues.apache.org/jira/browse= /AXIS2-2931?focusedCommentId=3D12676837#action_12676837


--001636c5aace08d4ff0463e2d782--