hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: Lockup in XFire
Date Tue, 08 Aug 2006 08:43:59 GMT
On Mon, 2006-08-07 at 18:10 -0700, Patrick Lightbody wrote:
> On Aug 7, 2006, at 3:18 PM, Oleg Kalnichevski wrote:
> 
> > On Mon, 2006-08-07 at 14:40 -0700, Patrick Lightbody wrote:
> >> Oleg,
> >> Sure, I understand that the issue is due to a native timeout - but
> >> the question is: why?
> >>
> >
> > (1) The only reason for a native socket read to block indefinitely is
> > the SO_TIMEOUT value set to zero. Does XFire explicitly set the socket
> > timeout value to a positive value?
> 
> No, XFire does not. I tried setting it and that didn't produce the  
> desireable affect, as I have several requests that I want to take up  
> to several minutes to respond. I think an indefinite time is fine  
> (which the default, which is what I was using). My issue isn't that  
> the timeout is set to 0, but that the connection isn't proceeding  
> like it should be. 

The connection is not proceeding because apparently the server is not
sending any data.

> Specifically, on the server side, I have done a  
> stack dump and the only thing related to sockets there is this:
> 
> ---> Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8081]
>      java.net.PlainSocketImpl.socketAccept(Native Method)
>      java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>      java.net.ServerSocket.implAccept(ServerSocket.java:450)
>      java.net.ServerSocket.accept(ServerSocket.java:421)
>      org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java: 
> 432)
>      org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java: 
> 634)
> 
> Which clearly is not related to the other open socket.
> 

Are there any other threads not related to sockets? The server may stop
responding on a connection if the connection thread gets dead-locked on
a mutex or enters an infinite loop.  

> So I'm back to my original problem: why is the client stuck in the  
> native state while the server has no record of it? And why does it  
> only happen for that one request in those exact steps that I execute.  
> I should also add that I have done a netstat call at the time, and  
> got back:
> 
> [hostedqa-j2ee@autoriginate01 webapps]$ netstat -a --numeric-port |  
> grep 8081
> tcp        1      0 ::ffff:192.168.167.1:58125  ::ffff: 
> 192.168.167.129:8081 CLOSE_WAIT
> 
> Any other ideas? I'm really at a loss here :(
> 

As far as I am concerned there's nothing wrong in how HttpClient is
behaving. It blocks indefinitely in the native socket read because this
is precisely what it has been instructed to do. In my opinion the
problem is on the server side. 

Just in case consider upgrading the JVM you are using both client and
server side to the latest patch level, and consider using a different
version of Jetty or a different servlet engine to see if that makes any
difference

Oleg


> Patrick
> 
> 
> > (2) org.apache.commons.httpclient.HttpMethodBase writeRequest
> > 100 (continue) read timeout. Resume sending the request
> >
> > This message is logged when the target server fails to properly  
> > respond
> > to the "Expect: 100-continue" handshake. When the handshake is  
> > activated
> > HttpClient sends the request header containing the "Expect:
> > 100-continue" directive prior to sending the request body and expects
> > the target server to respond with status code 100 indicating that  
> > it is
> > okay to proceed with sending the request content. Apparently the  
> > server
> > has issues with the "Expect: 100-continue" handshake or simply  
> > locks up
> > while processing the request header.
> >
> 
> I turned off this feature and the INFO log goes away, but the lock up  
> still occurs. So I don't think this is related.
> 
> > Hope this helps.
> >
> > Oleg
> >
> >> Like I reported in my initial email, subsequent requests in other
> >> threads (coming from test.jsp) work fine.
> >>
> >> Patrick
> >>
> >> Patrick Lightbody
> >> Autoriginate, Inc.
> >> 503-488-5402
> >> http://www.autoriginate.com
> >> patrick@autoriginate.com
> >>
> >> "Intelligent testing made convenient"
> >>
> >>
> >> On Aug 7, 2006, at 2:34 PM, Oleg Kalnichevski wrote:
> >>
> >>> On Mon, 2006-08-07 at 12:29 -0700, Patrick Lightbody wrote:
> >>>> I've tried using XFire 1.1.1 and 1.2-RC, combined with  
> >>>> HttpClient 3.0
> >>>> and 3.1-alpha1. I get the same result, outlined below, which  
> >>>> causes a
> >>>> complete lockup of a thread. I can't figure out what would cause
> >>>> this.
> >>>>
> >>>> When making a call via XFire (ClientService.getAppLog()), the  
> >>>> current
> >>>> thread locks up just after printing the following out in the logs:
> >>>>
> >>>> org.apache.commons.httpclient.HttpMethodBase writeRequest
> >>>> 100 (continue) read timeout. Resume sending the request
> >>>>
> >>>> I see that this log comes from an InterruptedIOException here:
> >>>>
> >>>> http://jakarta.apache.org/commons/httpclient/xref/org/apache/ 
> >>>> commons/
> >>>> httpclient/HttpMethodBase.html#2004
> >>>>
> >>>> The stack dump of the locked thread is:
> >>>>
> >>>> "Thread-62" daemon prio=1 tid=0x082602c0 nid=0x51ca runnable
> >>>> [0x79926000..0x79926e30]
> >>>
> >>> Patrick,
> >>> As you can see the thread gets blocked in the native socket read
> >>> method,
> >>> so this is very unlikely to be a threading dead-lock in the  
> >>> HttpClient
> >>> code. Most likely the socket read operation blocks indefinitely
> >>> because
> >>> socket timeout is not set (SO_TIMEOUT value is set to zero).
> >>>
> >>> Hope this helps
> >>>
> >>> Oleg
> >>>
> >>>
> >>>>         at java.net.SocketInputStream.socketRead0(Native Method)
> >>>>         at java.net.SocketInputStream.read(SocketInputStream.java:
> >>>> 129)
> >>>>         at java.io.BufferedInputStream.fill 
> >>>> (BufferedInputStream.java:
> >>>> 218)
> >>>>         at java.io.BufferedInputStream.read 
> >>>> (BufferedInputStream.java:
> >>>> 235)
> >>>>         - locked <0x830328c8> (a java.io.BufferedInputStream)
> >>>>         at org.apache.commons.httpclient.HttpParser.readRawLine
> >>>> (HttpParser.java:77)
> >>>>         at org.apache.commons.httpclient.HttpParser.readLine
> >>>> (HttpParser.java:105)
> >>>>         at org.apache.commons.httpclient.HttpConnection.readLine
> >>>> (HttpConnection.java:1115)
> >>>>         at
> >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager
> >>>> $HttpConnectionAdapter.readLine
> >>>> (MultiThreadedHttpConnectionManager.java:1373)
> >>>>         at
> >>>> org.apache.commons.httpclient.HttpMethodBase.readStatusLine
> >>>> (HttpMethodBase.java:1832)
> >>>>         at  
> >>>> org.apache.commons.httpclient.HttpMethodBase.readResponse
> >>>> (HttpMethodBase.java:1590)
> >>>>         at org.apache.commons.httpclient.HttpMethodBase.execute
> >>>> (HttpMethodBase.java:995)
> >>>>         at
> >>>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry
> >>>> (HttpMethodDirector.java:397)
> >>>>         at
> >>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod
> >>>> (HttpMethodDirector.java:170)
> >>>>         at org.apache.commons.httpclient.HttpClient.executeMethod
> >>>> (HttpClient.java:396)
> >>>>         at
> >>>> org.codehaus.xfire.transport.http.CommonsHttpMessageSender.send
> >>>> (CommonsHttpMessageSender.java:226)
> >>>>         at
> >>>> org.codehaus.xfire.transport.http.HttpChannel.sendViaClient
> >>>> (HttpChannel.java:118)
> >>>>         at org.codehaus.xfire.transport.http.HttpChannel.send
> >>>> (HttpChannel.java:48)
> >>>>         at org.codehaus.xfire.handler.OutMessageSender.invoke
> >>>> (OutMessageSender.java:26)
> >>>>         at org.codehaus.xfire.handler.HandlerPipeline.invoke
> >>>> (HandlerPipeline.java:130)
> >>>>         at org.codehaus.xfire.client.Invocation.invoke
> >>>> (Invocation.java:75)
> >>>>         at org.codehaus.xfire.client.Client.invoke(Client.java:335)
> >>>>         at org.codehaus.xfire.client.XFireProxy.handleRequest
> >>>> (XFireProxy.java:77)
> >>>>         at org.codehaus.xfire.client.XFireProxy.invoke
> >>>> (XFireProxy.java:57)
> >>>>         at $Proxy5.getAppLog(Unknown Source)
> >>>>         at com.hostedqa.model.TestContextImpl.dispose
> >>>> (TestContextImpl.java:83)
> >>>>         at com.hostedqa.model.Suite.playback(Suite.java:85)
> >>>>         at com.hostedqa.service.PlaybackService.runTest
> >>>> (PlaybackService.java:83)
> >>>>         at com.hostedqa.service.PlaybackService.playSuite
> >>>> (PlaybackService.java:48)
> >>>>         at com.hostedqa.action.project.suite.PlayAction$1.run
> >>>> (PlayAction.java:25)
> >>>>         at java.lang.Thread.run(Thread.java:595)
> >>>>
> >>>> What's very weird is that I am able to drop a JSP (test.jsp) that
> >>>> makes the exact same call and it completes just fine. This tells me
> >>>> that there is something environmental about _this_ thread that  
> >>>> causes
> >>>> HttpClient to do this. The call alone is not the issue.
> >>>>
> >>>> Also, I might add that the XFire call never makes it to the  
> >>>> other end
> >>>> (ClientServiceImpl), as I have a print line there that never gets
> >>>> invoked. I ran a stack dump on the other side as well, and nothing
> >>>> stood out (though it is possible part of the request made it  
> >>>> through
> >>>> to XFire's Servlet, and then broke and was no longer in the active
> >>>> thread dump by the time I forced the dump).
> >>>>
> >>>> Finally, this request is running over HTTP. I'd really like to  
> >>>> figure
> >>>> out:
> >>>>
> >>>> 1) What that log from HttpMethodBase.writeRequest() is all about
> >>>> 2) Why there would be a perpetual "pause" in the native method, but
> >>>> no actual visible deadlock.
> >>>> 3) How to fix this :)
> >>>>
> >>>> Patrick
> >>>>
> >>>> Patrick Lightbody
> >>>> Autoriginate, Inc.
> >>>> 503-488-5402
> >>>> http://www.autoriginate.com
> >>>> patrick@autoriginate.com
> >>>>
> >>>> "Intelligent testing made convenient"
> >>>>
> >>>>
> >>>>
> >>>> -------------------------------------------------------------------

> >>>> --
> >>>> To unsubscribe, e-mail: httpclient-user-
> >>>> unsubscribe@jakarta.apache.org
> >>>> For additional commands, e-mail: httpclient-user-
> >>>> help@jakarta.apache.org
> >>>>
> >>>>
> >>>
> >>>
> >>> -------------------------------------------------------------------- 
> >>> -
> >>> To unsubscribe, e-mail: httpclient-user- 
> >>> unsubscribe@jakarta.apache.org
> >>> For additional commands, e-mail: httpclient-user-
> >>> help@jakarta.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: httpclient-user- 
> >> unsubscribe@jakarta.apache.org
> >> For additional commands, e-mail: httpclient-user- 
> >> help@jakarta.apache.org
> >>
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: httpclient-user- 
> > help@jakarta.apache.org
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org


Mime
View raw message