tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From R Mundell <r_mund...@hotmail.com>
Subject Intermittently a Tomcat’s 7's response loses all (non-Coyote-added) headers
Date Thu, 03 May 2018 14:47:11 GMT
Attention Tomcat developers who know how the Coyote bit of Tomcat works… I’ve got a tricky
one for you! :-)

We’re struggling with a puzzling problem where intermittently, calls to a servlet are delivered
back to the client with all of the headers missing except the ones that Coyote adds.

In roughly 9,999 out of 10,000 attempts the response returns correctly (which includes a number
of headers which are added by filters, a body response which is gzipped by the servlet and
a “Content-Encoding” header which indicates it’s gzipped, and the correct “Content-Length”
header).

But, intermittently, approximately 1 time out of 10,000 requests, the response back to the
client contains only 3 headers: “Date”, “Server”, and a “Transfer-Encoding” header
set to “chunked” (even though the response body is not chunked). The result of this is
that the gzipped body is presented back to the client with the “Content-Encoding” header
missing so it’s rendered unreadable by the browser, and all of the other headers are missing.
(We’ve determined this by being able to capture a couple of instances of this issue in Fiddler).

Looking at the Coyote source code it would appear that if the “Content-Length” header
isn’t presented to Coyote from the upstream handling of the filters and servlet then it’ll
default to a chunked encoding, which explains why the response is getting chunked!

So, basically - the filters run without error, the servlet runs without error, but the when
the response is received back into Coyote’s AbstractHttp11Processor.process() method, the
headers have disappeared!

Annoyingly we can’t find a recreatable test case, but anecdotally it’s when the server
is heavily loaded, but there are still plenty of pooled workers available, and plenty of memory.
This is a very high volume server (probably millions of requests a day) so adding verbose
logging within Coyote isn’t a realistic option.

I don’t believe it’s actually possible for the filters or the servlet to remove headers
even if they want to (they don’t appear to have any access to the MimeHeaders of the Coyote
“Response” object, and the HttpServletResponse object doesn’t have any methods which
allow header removal).

The only clue we have is occasionally we see “INFO: Encountered a non-recycled response
and recycled it forcedly.” in the logs, but these errors are infrequent compared to the
number of instances of the problem, and the timestamps of these errors don’t correspond
to when users have reported the error.

My best guess would be somehow the Coyote “Request” object is accidentally accessed by
two threads concurrently to service two requests, and one request is calling the Request.recycle()
(which is calling headers.recycle() which removes all of the headers) but I’m struggling
to see how that could happen. Note we are not using async servlets.

I appreciate 7.0.72 isn’t the very latest and greatest version but I can’t find anything
in the change log for later 7.x, 8.x, 8.5.x or 9.x releases which indicate this is a known
issue or resolved.

Any ideas about how to further diagnose this would be very gratefully received. Otherwise
our next step is to blindly update to later versions and just hope it goes away.

Thanks,

Richard

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message