tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reich, Matthias" <matthias.re...@siemens.com>
Subject RE: Memory Leak with Comet
Date Thu, 26 Apr 2007 10:19:24 GMT
Excuse me that I left you alone during the discussion yesterday!

Let me clarify: The scenario is that the client polls for events.
Thus, it sends a subsequent poll request once it got an answer from the server.
(And how should I bring Tomcat to create hundreds of request objects with four browsers, if
each of them issues only a single request?)

In the log file I can see a very regular pattern (where request parameter count=80 is the
client side request count
and (80) s the server side request count)

25.04.2007 19:41:53 comettest.CometServlet event
WARNUNG: BEGIN(80) POST /comettest/comet/request?action=poll&count=80
25.04.2007 19:41:53 comettest.CometServlet$EventProvider run
WARNUNG: queue size: 1
25.04.2007 19:41:53 comettest.CometServlet$EventProvider sendResponse
WARNUNG: RESPONSE(80)
25.04.2007 19:41:53 comettest.CometServlet closeEvent
WARNUNG: CLOSE(80) by response provider
25.04.2007 19:41:53 comettest.CometServlet$EventProvider run
WARNUNG: queue size: 0
25.04.2007 19:41:53 comettest.CometServlet$EventProvider run
WARNUNG: sleeping 750 millis
25.04.2007 19:41:54 comettest.CometServlet event
WARNUNG: END(80) POST /comettest/comet/request?action=poll&count=80
25.04.2007 19:41:54 comettest.CometServlet closeEvent
WARNUNG: CLOSE(80) by event processor
25.04.2007 19:41:54 comettest.CometServlet event
WARNUNG: BEGIN(81) POST /comettest/comet/request?action=poll&count=81

Now I look at this statement from a former post again:
> Processors are only recycled when the IO layer gets an event (read,
> disconnect, timeout), and if:
> - the event is a non timeout error
> - the event is closed by the servlet
> - the event has been closed asynchronously

I must admit that there is no read, disconnect, or timeout event - but there is an END event,
and as the CoyoteAdapter sets the type to END instead READ because the event was closed before,
I think the mentioned precondition for recycling is fulfilled.

The servlet was able to process 467 requests before the OutOfMemoryError occurred,
but in the memory dump I can see only 103 processor instances, thus the majority of the Processors
was in fact recycled.
The mismatch might result from the fact the browser sometimes closed the connection in between,
and sometimes did not - see below.

If I replace the event.close() in the response provider thread with an outputStream.close(),
I also get an END event, regardless if I set Connection: close or not, and also with the NIOConnector.

But there is a difference: If the response provider thread sets the Connection: close header
before closing the stream,
memory consumption is fairly stable for a long time (I let it run for over an hour and saw
no significant increase in the Windows task manager), so let us assume that recycling works
fine in that situation.


MY CONCLUSION: I have to closely couple the connection lifecycle with the request lifecycle
to make things work.


However, this is bad news for me - I would like to keep connections open as long as possible,
because especially with an SSL connector opening a new connection after each poll request
is expensive.
(And what about malicious clients that ignore the Connection: close ?)

If I try to keep the connection open (i.e. the content length in the response header and flush
the OutputStream, but don't close it),
the client sends the subsequent request on the same connection.
The RequestProcessor for the old Request (yes, Request and not Connection - that's why it
is called RequestProcessor and not ConnectionProcessor) issues a READ event in this case.
The Servlet tries to read and gets -1. This means that it does not actually read from the
connection - otherwise it would read the unparsed subsequent request. (We could say that,
upon a READ event, the servlet reads further portions of the request, it does not simple read
from the connection).
As it got -1, the Servlet calls event.close() which means: I am done with this request (it
does not mean: I am done with the connection).
Then a new (or a recycled) RequestProcessor is associated with the same connection and processes
the BEGIN event for the new request, but it seems as if the old RequestProcessor is not always
recycled.



I think that recycling could be organized much clearer with a small change in the execution
model which would also have the benefit
of descibing the lifecycle of asynchronous requests more clearly:

- First of all, I would like to rename CometEvent into CometRequest, and let it be part of
the contract between container and Servlet that one instance of this class represents a request
during the whole request lifecycle.
(I.e. the Servlet is allowed to hold references to CometRequest objects for further processing)

- Recycling of a CometRequest (and all associated objects including the RequestProcessor)
should be shifted completely to the CometRequest.close() method.

- The only restriction in that case would be that CometRequest.close then cannot be invoked
from within the CometProcessor.event() method (Obviously the RequestProcessor cannot recycle
itself while it is processing a request).

For regular request processing CometRequest.close will be invoked by a different thread anyway.
There could be some overhead for handling ERROR events as these must be passed to another
thread for closing the CometRequest,
but hopefully we could get rid of most of such situations with that change.
E.g. I would expect that a READ event after sending the response (and keeping the connection
alive)
would no longer happen if the sender of the response uses CometRequest.close, because the
RequestProcessor would then be detached from the connection early enough.

There would surely be a need for READ events to enable the Servlet to trigger further processing
by other threads
(i.e. another thread would actually read from the input stream and react to an IOException
or a -1 response).

ERROR events will be required to enable the Servlet to trigger some cleanup in another thread
(e.g. remove the CometRequest instance from internal queues ...), including a call to CometRequest.close.


The clear semantics of this approach would be: Processing is the same as for synchronous requests
, except that the RequestProcessor thread is released from processing early, and another thread
(or several threads that synchronize when accessing the request) continues processing at any
time it wants, and the application fully controls the lifetime of CometRequest instances.


Matthias


-----Original Message-----
From: Rémy Maucherat [mailto:remy.maucherat@gmail.com] 
Sent: Wednesday, April 25, 2007 6:42 PM
To: Tomcat Users List
Subject: Re: Memory Leak with Comet

On 4/25/07, Filip Hanik - Dev Lists <devlists@hanik.com> wrote:
> then it's normal behavior, he'll just have to wait for a timeout or the
> client will disconnect

I don't know for sure, of course, but it's my theory at the moment.

Rémy


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message