tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Thomas <ma...@apache.org>
Subject Re: Tomcat occasionally duplicating responses
Date Mon, 18 May 2015 10:22:19 GMT
On 18/05/2015 10:31, Stephen Dawkins wrote:
> Hi
> 
> I have an application that uses embedded Tomcat (8.0.22) to serve SOAP
> services. I've recently attempted to add javamelody[1] to the app to gather
> some stats, however I've run into an issue where occasionally the response
> is incorrect, mostly it just duplicates response, although sometimes it's
> not a valid HTTP response. I can only seem to reproduce this on a Linux box
> (which is running Fedora 19), my Windows machine doesn't appear to have the
> same issue.
> 
> I've confirmed that the application code is only sending the response once,
> and the error only occurs when using the javamelody servlet filter.
> However, investigating the filter didn't reveal anything obvious that would
> cause the problem (it just collects the amount of bytes sent). It does
> override the flushBuffer method in the class it uses to wrap the
> HttpServletResponse, but I get the same error even when I don't explicitly
> call that method from the application code.

Tomcat will call flushBuffer() internally.

Tomcat does re-use Request/Response objects so if a reference is
accidentally retained to one of them you can see this issue.

Try using:
org.apache.catalina.connector.RECYCLE_FACADES=true

(not the best named attribute but true will cause a new facade to be
created for every request/response).

> The client I'm using to test with just uses Apache CXF to send one request
> at a time to the server. Interestingly I couldn't seem to reproduce it when
> I increased the amount of threads that the client was using, however it's
> not the most reliable error to reproduce with just a single thread.
> 
> I've tried setting breakpoints in various places in the application,
> javamelody and Tomcat, but I couldn't get any useful information out of it
> (since I'm having to debug remotely, the conditional breakpoints are slow
> enough to make reproducing it extremely difficult), and I'm not entirely
> certain where the issue is. The only information I've managed to glean is
> from using strace to capture the syscalls:
> 
> pid   timestamp       syscall
> 32189 16:19:34.330576 write(118, "HTTP/1.1 200 OK\r\nServer:
> Apache-Coyote/1.1"..., 149) = 149
> 32189 16:19:34.331728 write(118, "51e\r\n", 5) = 5
> 32189 16:19:34.332788 write(118, "<?xml version=\"1.0\"
> encoding=\"UTF-8\""..., 1310) = 1310
> 32189 16:19:34.335779 write(118, "\r\n", 2) = 2
> 32216 16:19:34.336863 write(118, "51e\r\n", 5) = 5
> 32216 16:19:34.337734 write(118, "<?xml version=\"1.0\"
> encoding=\"UTF-8\""..., 1310) = 1310
> 32216 16:19:34.344520 write(118, "\r\n", 2) = 2
> 32200 16:19:34.346986 write(118, "0\r\n\r\n", 5) = 5
> 
> The pid column is the most interesting. This shows that there are 2
> separate threads sending the same response to the client, which doesn't
> seem right. Occasionally, I get a slightly different error in the client
> (the above response just produces an XML parse error):

Are those both Java threads from the same JVM?

Can you correlate seeing multiple threads with the error occurring?

> 32201 16:17:29.592419 write(161, "HTTP/1.1 200 OK\r\nServer:
> Apache-Coyote/1.1"..., 149) = 149
> 32201 16:17:29.593717 write(161, "51e\r\n", 5) = 5
> 32201 16:17:29.594848 write(161, "<?xml version=\"1.0\"
> encoding=\"UTF-8\""..., 1310) = 1310
> 32235 16:17:29.597871 write(161, "51e\r\n", 5) = 5
> 32235 16:17:29.603263 write(161, "<?xml version=\"1.0\"
> encoding=\"UTF-8\""..., 1310) = 1310
> 32235 16:17:29.603977 write(161, "\r\n", 2) = 2
> 32201 16:17:29.605333 write(161, "\r\n", 2) = 2
> 32216 16:17:29.608646 write(161, "0\r\n\r\n", 5) = -1 EPIPE (Broken pipe)
> 
> This produces the following error:
> 
> java.io.IOException: missing CR
>         at
> sun.net.www.http.ChunkedInputStream.processRaw(ChunkedInputStream.java:405)
>         at
> sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:572)
>         ...
> 
> Again, 2 separate threads, but this time interleaved in a way that's
> invalid.
> 
> While I'm happy enough to fix any issue in javamelody if that is what is
> causing it, I'm very concerned that it's uncovered a latent issue in Tomcat.

Unlikely but possible to be a Tomcat issue with the information we have
so far.

> Does anyone have any suggestions as to where the issue could be?

Are you using any Servlet 3.0 async features?

Mark

> 
> [1] https://code.google.com/p/javamelody/
> 
> Thanks & Regards
> Stephen
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message