Return-Path: Delivered-To: apmail-ws-axis-dev-archive@www.apache.org Received: (qmail 75147 invoked from network); 27 Feb 2009 11:20:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 27 Feb 2009 11:20:18 -0000 Received: (qmail 3645 invoked by uid 500); 27 Feb 2009 11:20:15 -0000 Delivered-To: apmail-ws-axis-dev-archive@ws.apache.org Received: (qmail 3562 invoked by uid 500); 27 Feb 2009 11:20:15 -0000 Mailing-List: contact axis-dev-help@ws.apache.org; run by ezmlm Precedence: bulk Reply-To: axis-dev@ws.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list axis-dev@ws.apache.org Received: (qmail 3552 invoked by uid 99); 27 Feb 2009 11:20:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 03:20:15 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kdobrik.axis2@googlemail.com designates 209.85.218.171 as permitted sender) Received: from [209.85.218.171] (HELO mail-bw0-f171.google.com) (209.85.218.171) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 11:20:04 +0000 Received: by bwz19 with SMTP id 19so1025042bwz.16 for ; Fri, 27 Feb 2009 03:19:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=MdyDyAVkOtDy6NwHp7re+602vbHHTlBLYhFbfnOHi2A=; b=IAgsMeIaYYf7RcWs5Df6DMfYRmcUn3zs8hKexUt2f/ukPsr93qX0ist7IjTyLEpDvb louak9rm17bx9J6ptQaQkMhZMlRf5LDQ3UiDqAGf4oXTMmfuPLIJ/J/1DL5WZi0W8cF1 PmKbFIdyMWVmXLD61Esi1VNmuIm5KCm3kbsrs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=DoGHNsqWnh9CxUkFEr5pH74h/2ObDP3eCykGr185h+O5ltZUUE0amvuUXflfqMugZ6 zJQ6qETDj43v91N81qDgXMfKvKfpiK9z6xQ8+9l7YtIMVPEiVx8ZyCqfPIzEafefJ4zm 6+rg8HjXnAvi6z/M8qPEjK8U8mF/bdKpZz2Iw= MIME-Version: 1.0 Received: by 10.180.242.18 with SMTP id p18mr824772bkh.173.1235733583848; Fri, 27 Feb 2009 03:19:43 -0800 (PST) In-Reply-To: References: <3b3bfaf80902270110v453110bal126c5d4930267797@mail.gmail.com> Date: Fri, 27 Feb 2009 13:19:43 +0200 Message-ID: <3b3bfaf80902270319j19310719h2fa2bf1f0ce2d3ec@mail.gmail.com> Subject: Re: HTTP connection leak and other related issues From: Dobri Kitipov To: axis-dev@ws.apache.org Content-Type: multipart/alternative; boundary=001636c5b5984d65d80463e4a539 X-Virus-Checked: Checked by ClamAV on apache.org --001636c5b5984d65d80463e4a539 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi Andreas, thank you for the comment. I think you get the question. Quick test shows that setting the following line of code into the client: options.setCallTransportCleanup(true); forces the closure of the http connection. It seems it is not the default behavior. This is a good and fast solution. I was a little bit more focused on wondering why I have such a difference b/n using SOAP and MIME builder. I need to think about some use cases when we need to have options.setCallTransportCleanup(false). Can we have this by default in some cases? Anyway, it will be worth having a further analysis of the issue we have with SOAPBuilder behavior. Thank you, Dobri On Fri, Feb 27, 2009 at 12:46 PM, Andreas Veithen wrote: > If I understand correctly, Dobri's findings can be summarized as follows: > 1. Once the InputStream is consumed, commons-httpclient automatically > releases the connection. > 2. SOAPBuilder never completely consumes the InputStream. > > The SOAPBuilder behavior is indeed somewhat questionable, but it is > important to understand that because of the deferred parsing model > used by Axiom, there is never a guarantee that the InputStream will be > completely consumed. Normally releasing the connection is the > responsibility of the CommonsHTTPTransportSender#cleanup method which > should be called by ServiceClient#cleanupTransport. It would be > interesting to know if that method is called, and if it is, why it > fails to release the connection. > > Andreas > > On Fri, Feb 27, 2009 at 10:10, Dobri Kitipov > wrote: > > Hi all, > > I have observed absolutely the same thing these days. I need some more > time > > to analyze the whole picture, but here is my current synthesis of the > issue. > > > > It seems that http connection release is tightly coupled with the Axis2 > > builder used to handle and process the response body and the > corresponding > > input streams used. The builder and the InputStream used are based on the > > HTTP headers fields like "Content-Type" and "Transfer-Encoding" (e.g. > > Content-Type: text/xml; charset=UTF-8 Transfer-Encoding: chunked). So if > we > > have Content-Type: text/xml; then SOAPBuilder class will be used. If we > > have type="application/xop+xml" then MIMEBuilder will be used. > > > > The successfull story when we have MIMIBuilder: > > > > When MIMEBuilder is used then the response Buffered InputStream (IS) is > > wrrapped (I will use "->" sign as substitution for wrrapped) ChunkedIS -> > > AutoCloseIS (this has a responseConsumed() method notified when IS.read() > > returns -1, which means that the response IS has been completely read) -> > > PushBackIS ->BounderyPushBackIS. The BounderyPushBackIS reads the > response > > stream (see readFromStream(....)) in a cycle till it reaches its end. At > > every iteration of this cycle a AutoCloseIS checkClose(l) is invoked. So > > when the end is reached (-1 is returned) then this check causes the > > invokation of the AutoCloseIS checkClose(...) method. This method > invokes > > notifyWatcher() that in turn invokes responseBodyConsumed() method of the > > HttpMethodBase class. This causes the release of the http connection > which > > is returned back to the connection pool. So here we have no problem with > > connection reuse. > > > > The bad story we have with SOAPBuilder: > > > > When SOAPBuilder is used then the response Buffered InputStream is > wrrapped > > in a ChunkedIS -> AutoCloseIS -> PushBackIS. May be you has noticed that > > BounderyPushBackIS is not used. As a result the respose IS is not > completely > > read (in fact this is not really correct, it could be read but the > > invokation of the PushBackIS unread(...) method causes the AutoCloseIS > > checkClose() method never to return -1). As a result the http connection > is > > not released. And since there is a limit to have 2 connection per host > > then after the second invokation of the WS client the thread hangs > waiting > > for a free connection. > > > > Please, provide us with your comments on this issue. > > > > Thank you in advance. > > > > Regards, > > Dobri > > > > On Fri, Feb 27, 2009 at 3:54 AM, Alexis Midon wrote: > >> > >> no taker for an easy patch? > >> > >> Alexis > >> > >> > >> On Wed, Feb 25, 2009 at 6:45 PM, Alexis Midon > wrote: > >>> > >>> Hi everyone, > >>> > >>> All the issues relatives to AXIS2-935 are really messy, some of them > are > >>> closed but their clones are not. Some are flagged as fixed but are > obviously > >>> not. All these issues are really old, so I'd like to take a chance to > bring > >>> them back to your attention, especially before releasing 1.5. > >>> > >>> I'll post a description of the issue in this email as a summary all the > >>> jiras. > >>> > >>> By default, ServiceClient uses one HttpConnectionManager per invocation > >>> [2]. This connection manager will create and provide one connection to > >>> HTTPSender. The first issue is that by default this connection is never > >>> released to the pool [3]. if you do zillions of invocations, this leak > will > >>> max out your number of file descriptors. > >>> > >>> Your investigations in Axis2 options quickly lead you to the > >>> REUSE_HTTP_CLIENT option. But this first issue has some unfortunate > >>> consequences if you activate it. Actually if you do so, a single > connection > >>> manager is shared across all invocations. But because connections are > not > >>> release, the pool is starved after two invocations, and the third > invocation > >>> hangs out indefinitely. :( > >>> > >>> If you keep digging you will find the AUTO_RELEASE_CONNECTION option. > Its > >>> sounds like a good lead! Let's try it. If you activate this option the > >>> connection is properly released -Yahoooo! the leak is fixed - but > >>> unfortunately a new issue shows up (issue #2, aka AXIS2-3478). > >>> AbstractHTTPSender passes the stream of the connection to the message > >>> context [4] , but that the connection is now properly released, so this > >>> stream is closed before the SOAPBuilder gets a chance to read the > response > >>> body. > >>> Boom! "IOException: Attempted read on closed stream" > >>> > >>> These issues are easily reproducible in versions 1.3, 1.4, 1.5. > >>> > >>> I submitted and documented a fix in AXIS2-2931 [5], if you had a chance > >>> to look at it that would be much appreciate. > >>> > >>> > >>> Alexis > >>> > >>> > >>> [1] > >>> > https://issues.apache.org/jira/browse/AXIS2-935?focusedCommentId=12513543#action_12513543 > >>> [2] see method getHttpClient in > >>> > https://svn.apache.org/repos/asf/webservices/commons/trunk/modules/transport/modules/http/src/org/apache/axis2/transport/http/AbstractHTTPSender.java > >>> [3] see method cleanup in > >>> > https://svn.apache.org/repos/asf/webservices/commons/trunk/modules/transport/modules/http/src/org/apache/axis2/transport/http/HTTPSender.java > >>> [4] see method processResponse in AbstractHTTPSender.java > >>> [5] > >>> > https://issues.apache.org/jira/browse/AXIS2-2931?focusedCommentId=12676837#action_12676837 > >> > > > > > --001636c5b5984d65d80463e4a539 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Andreas,
thank you for the comment. I think you get the question. Quick test shows that setting the following line of code into the client:<= br>
options.setCallTransportCleanup(true);

forces the closure of= the http connection. It seems it is not the default behavior. This is a go= od and fast solution. I was a little bit more focused on wondering why I ha= ve such a difference b/n using SOAP and MIME builder.

I need to think about some use cases when we need to have options.setCa= llTransportCleanup(false). Can we have this by default in some cases?
Anyway, it will be worth having a further analysis of the issue we have w= ith SOAPBuilder behavior.

Thank you,
Dobri


On Fri, Feb = 27, 2009 at 12:46 PM, Andreas Veithen <andreas.veithen@gmail.com> wrot= e:
If I understand c= orrectly, Dobri's findings can be summarized as follows:
1. Once the InputStream is consumed, commons-httpclient automatically
releases the connection.
2. SOAPBuilder never completely consumes the InputStream.

The SOAPBuilder behavior is indeed somewhat questionable, but it is
important to understand that because of the deferred parsing model
used by Axiom, there is never a guarantee that the InputStream will be
completely consumed. Normally releasing the connection is the
responsibility of the CommonsHTTPTransportSender#cleanup method which
should be called by ServiceClient#cleanupTransport. It would be
interesting to know if that method is called, and if it is, why it
fails to release the connection.

Andreas

On Fri, Feb 27, 2009 at 10:10, Dobri Kitipov
<kdobrik.axis2@googlemai= l.com> wrote:
> Hi all,
> I have observed absolutely the same thing these days. I need some more= time
> to analyze the whole picture, but here is my current synthesis of the = issue.
>
> It seems that http connection release is tightly coupled=A0 with the A= xis2
> builder used to handle and process the response body and the correspon= ding
> input streams used. The builder and the InputStream used are based on = the
> HTTP headers fields like "Content-Type" and "Transfer-E= ncoding" (e.g.
> Content-Type: text/xml; charset=3DUTF-8=A0 Transfer-Encoding: chunked)= . So if we
> have Content-Type: text/xml; then SOAPBuilder class will be used. If w= e
> have=A0 type=3D"application/xop+xml" then MIMEBuilder will b= e used.
>
> The successfull story when we have MIMIBuilder:
>
> When MIMEBuilder is used then the response Buffered InputStream (IS) i= s
> wrrapped (I will use "->" sign as substitution for wrrapp= ed) ChunkedIS ->
> AutoCloseIS (this has a responseConsumed() method notified when IS.rea= d()
> returns -1, which means that the response IS has been completely read)= ->
> PushBackIS ->BounderyPushBackIS. The BounderyPushBackIS reads the r= esponse
> stream (see readFromStream(....)) in a cycle till it reaches its end. = At
> every iteration of this cycle a AutoCloseIS checkClose(l) is invoked. = So
> when the end is reached (-1 is returned) then this check causes the > invokation of the AutoCloseIS checkClose(...)=A0 method. This method i= nvokes
> notifyWatcher() that in turn invokes responseBodyConsumed() method of = the
> HttpMethodBase class. This causes the release of the http connection w= hich
> is returned back to the connection pool. So here we have no problem wi= th
> connection reuse.
>
> The bad story we have with SOAPBuilder:
>
> When SOAPBuilder is used then the response Buffered InputStream is wrr= apped
> in a ChunkedIS -> AutoCloseIS -> PushBackIS. May be you has noti= ced that
> BounderyPushBackIS is not used. As a result the respose IS is not comp= letely
> read (in fact this is not really correct, it could be read but the
> invokation of the PushBackIS unread(...) method causes the AutoCloseIS=
> checkClose() method never to return -1). As a result the http connecti= on is
> not released. And since there is a limit to have 2 connection per host=
> then after the second invokation of the WS client the thread hangs wai= ting
> for a free connection.
>
> Please, provide us with your comments on this issue.
>
> Thank you in advance.
>
> Regards,
> Dobri
>
> On Fri, Feb 27, 2009 at 3:54 AM, Alexis Midon <midon@intalio.com> wrote:
>>
>> no taker for an easy patch?
>>
>> Alexis
>>
>>
>> On Wed, Feb 25, 2009 at 6:45 PM, Alexis Midon <midon@intalio.com> wrote:
>>>
>>> Hi everyone,
>>>
>>> All the issues relatives to AXIS2-935 are really messy, some o= f them are
>>> closed but their clones are not. Some are flagged as fixed but= are obviously
>>> not. All these issues are really old, so I'd like to take = a chance to bring
>>> them back to your attention, especially before releasing 1.5.<= br> >>>
>>> I'll post a description of the issue in this email as a su= mmary all the
>>> jiras.
>>>
>>> By default, ServiceClient uses one HttpConnectionManager per i= nvocation
>>> [2]. This connection manager will create and provide one conne= ction to
>>> HTTPSender. The first issue is that by default this connection= is never
>>> released to the pool [3]. if you do zillions of invocations, t= his leak will
>>> max out your number of file descriptors.
>>>
>>> Your investigations in Axis2 options quickly lead you to the >>> REUSE_HTTP_CLIENT option. But this first issue has some unfort= unate
>>> consequences if you activate it. Actually if you do so, a sing= le connection
>>> manager is shared across all invocations. But because connecti= ons are not
>>> release, the pool is starved after two invocations, and the th= ird invocation
>>> hangs out indefinitely. :(
>>>
>>> If you keep digging you will find the AUTO_RELEASE_CONNECTION = option. Its
>>> sounds like a good lead! Let's try it. If you activate thi= s option the
>>> connection is properly released -Yahoooo! the leak is fixed - = but
>>> unfortunately a new issue shows up (issue #2, aka AXIS2-3478).=
>>> AbstractHTTPSender passes the stream of the connection to the = message
>>> context [4] , but that the connection is now properly released= , so this
>>> stream is closed before the SOAPBuilder gets a chance to read = the response
>>> body.
>>> Boom! "IOException: Attempted read on closed stream"=
>>>
>>> These issues are easily reproducible in versions 1.3, 1.4, 1.5= .
>>>
>>> I submitted and documented a fix in AXIS2-2931 [5], if you had= a chance
>>> to look at it that would be much appreciate.
>>>
>>>
>>> Alexis
>>>
>>>
>>> [1]
>>> https://issues.= apache.org/jira/browse/AXIS2-935?focusedCommentId=3D12513543#action_1251354= 3
>>> [2] see method getHttpClient in
>>> https://svn.apache.org/repos/asf= /webservices/commons/trunk/modules/transport/modules/http/src/org/apache/ax= is2/transport/http/AbstractHTTPSender.java
>>> [3] see method cleanup in
>>> https://svn.apache.org/repos/asf/webserv= ices/commons/trunk/modules/transport/modules/http/src/org/apache/axis2/tran= sport/http/HTTPSender.java
>>> [4] see method processResponse in AbstractHTTPSender.java
>>> [5]
>>> https://issues= .apache.org/jira/browse/AXIS2-2931?focusedCommentId=3D12676837#action_12676= 837
>>
>
>

--001636c5b5984d65d80463e4a539--