hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: Gracefully handling half-closed connections (encore!)
Date Wed, 15 Apr 2009 09:35:19 GMT
On Tue, Apr 14, 2009 at 11:39:01PM +0100, Sam Crawford wrote:
> Oleg,
> 
> Thanks for the quick reply, as always.
> 
> I'm afraid I haven't been able to get a wire log of the extreme scenario I
> highlighted earlier (I have a tcpdump capture of it from last night, but I
> doubt that's of much use to you).
> 
> I have however got a wire log of the issue as it's manifesting itself this
> evening. Inspection of the tcpdump traces reveal that this particular
> webserver is sending FINs for idle TCP connections after 30 seconds, whereas
> many of the other servers we're dealing with are all much higher (in the
> order of 5 minutes or so). This server does not reply with a Keep-Alive
> header, despite Connection: Keep-Alive being sent as a request header (I
> appreciate it's under no obligation to obey the client's wishes).
> 
> I've attached the wire log anyway, but I think for the moment I'm going to
> have to reduce my IdleConnectionEvictor to run at least every 30 seconds (it
> runs every two minutes at present).
> 
> A couple of closing questions if I may:
> 
> 1) Is there any way with the ClientConnectionManager to perform
> closeIdleConnections only for a specific HttpRoute? If the answer is
> "override the closeIdleConnections method and implement it" then that's fine
> by me :-)

A better approach could be implementing a custom
ConnectionKeepAliveStrategy and setting a lower keep-alive timeout for
known naughty servers.


> 2) You said the Stale Connection check is (in most cases) "evil". Is there
> any docs detailing exactly what it does and what we will lose if we disable
> it? (I believe it's enabled by default). The tiny speed hit is not a massive
> issue for us at the moment (but could be later).
> 

Stale connection checking simply cannot be 100% reliable as there is
always a window of time between a successful stale check and the request
execution when the connection can go stale on unsuspecting HttpClient.
A well designed application has to have a recovery code for such
situations anyways, which kind of makes stale connection checking pretty
much pointless.

The performance is not that tiny. 30 milliseconds for small payloads
that take only 3-5 milliseconds to execute is a lot.

Oleg

> Thanks again,
> 
> Sam
> 
> 
> 
> 2009/4/14 Oleg Kalnichevski <olegk@apache.org>
> 
> > Sam Crawford wrote:
> >
> >> Afternoon all,
> >> A few months back we had an issue with handling half closed TCP
> >> connections
> >> with HttpClient, and at the time I was advised to include something akin
> >> to
> >> the IdleConnectionEvictor - which we did and it's working very nicely in
> >> nearly all scenarios.
> >>
> >> However, in the past few days we've encountered a few WebLogic based hosts
> >> that aren't playing fair.
> >>
> >> The following is one (extreme) example of the issue we're encountering:
> >>
> >> Time (ms)    TCP action
> >> 0.0000         Client > Server [SYN]
> >> 0.5634         Server > Client [SYN,ACK]
> >> 1.2092         Client > Server [ACK]          <-- TCP session established
> >> 312.5276         Server > Client [FIN,ACK]
> >> 313.1309         Client > Server [ACK]
> >> 401.5089         Client > Server [HTTP POST /blah]
> >> 403.2986         Server > Client [RST]
> >>
> >> In the above example, the server closes its side of the connection only
> >> 300ms after establishment (by sending the FIN). (As an aside I'm curious
> >> as
> >> to why HttpClient is taking 400ms after the TCP connection has been
> >> established to send the request - any suggestions are also much
> >> appreciated,
> >> but this doesn't happen often).
> >>
> >>
> > This does not sound right. The stale connection check may cause a 20 to 30
> > millisecond delay (and generally should be avoided) but this is a bit too
> > much. Can you produce a wire / context log of the session?
> >
> >
> >  But the above is an extreme example. We see other cases where the WebLogic
> >> server is closing the connection of a keep-alive connection around 10-15
> >> seconds after the last request.
> >>
> >
> > Does the server send a 'Keep-alive' header with the response?
> >
> >
> >  Our IdleConnectionEvictor doesn't run that
> >
> >> often, so we end up with unusable connections. We could just run
> >> IdleConnectionEvictor more often, but that's not really desirable.
> >>
> >> I'm going to be digging into the WebLogic side of things this afternoon
> >> (to
> >> see if there's any limits we can modify there), but it does seem as though
> >> there should be a nice way for HttpClient to detect such cases. I've got
> >> stale connection checking enabled already by the way.
> >>
> >>
> > Stale connection checking is (in most cases) evil and should be avoided.
> >
> >  I'm interested in any feedback/ideas here! I can include a wire capture as
> >> an example if it would be helpful.
> >>
> >>
> > A wire / context log that correlates with the TCP dump would be great.
> >
> > Oleg
> >
> >  Thanks again,
> >>
> >> Sam
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >


> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message