tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastiaan van Erk <sebs...@sebster.com>
Subject Re: Exception with Comet in Tomcat with high load
Date Tue, 01 May 2007 16:16:05 GMT
Hi,

Did you synchronize the event.close() with same lock you use to write to 
the output stream (the SynchronizedOuputStream instance)?

I saw exactly the same stack trace you are getting due to (incorrect) 
concurrent access to the CoyoteOutputStream. The NullPointerException 
suggest that the request has already been cleaned up/recycled and yet 
you still write to it. Make sure that when you close the event, either 
the write becomes a null operation, or better, does not get called. It 
is a bug to call write on the response output stream after the event has 
been closed.

With the SynchronizedOutputStream construction I am relying on Tomcat 
not to close or call any methods on the CoyoteOutputStream outside of 
the CometProcessor's event() method. I don't know if this is correct (it 
has worked for me in practice so far, and I've been doing some heavy 
duty testing); however I do not know what else I could synchronize on. 
If anybody knows a better approach, I would like to know. :-)

Regards,
Sebastiaan

Martin Perez wrote:
> Hi,
>
> Thanks for the tip Sebastian, but it doesn't work for me. I added your
> class, wrapped all the output streams and synchronized close events 
> but the
> result is the same.
>
> Exception in thread "MessageSender[]" java.lang.NullPointerException
>        at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
> InternalNioOutputBuffer.java:607)
>        at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
> InternalNioOutputBuffer.java:600)
>        at org.apache.coyote.http11.Http11NioProcessor.action(
> Http11NioProcessor.java:1032)
>        at org.apache.coyote.Response.action(Response.java:181)
>        at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
> InternalNioOutputBuffer.java:571)
>        at org.apache.coyote.Response.doWrite(Response.java:560)
>        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
> OutputBuffer.java:353)
>        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java
> :434)
>        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
>        at org.apache.catalina.connector.OutputBuffer.writeBytes(
> OutputBuffer.java:381)
>        at org.apache.catalina.connector.OutputBuffer.write(
> OutputBuffer.java:370)
>        at org.apache.catalina.connector.CoyoteOutputStream.write(
> CoyoteOutputStream.java:89)
>        at org.apache.catalina.connector.CoyoteOutputStream.write(
> CoyoteOutputStream.java:83)
>        at com.cognotec.streaming.SynchronizedOutputStream.write(
> SynchronizedOutputStream.java:25)
>        at com.cognotec.streaming.CometRateSender.run(CometRateSender.java
> :86)
>        at java.lang.Thread.run(Thread.java:595)
>
> (As I said, I also synchronized all the close method calls either in 
> stream
> either in comet event)
>
> So, the question is: Is this a bug or simply I have reached the 
> scalability
> limit for my box?
>
> Cheers,
> Martin
>
> On 4/30/07, Sebastiaan van Erk <sebster@sebster.com> wrote:
>>
>> Hi,
>>
>> I had the same problems. It was a synchronization issue with me. What I
>> did was make a synchronized output stream wrapper (see attached file)
>> which wraps all access to the output stream, and I synchronized on this
>> output stream whenever I called event.close() on the Comet event
>> (especially in the READ event). When doing serveral IO operations on the
>> output stream which should not be interrupted, you can simply
>> synchronize on it (since SynchronizedOutputStream locks on itself).
>>
>> Regards,
>> Sebastiaan
>>
>>
>> Martin Perez wrote:
>> > Hi all,
>> >
>> > I have been testing Tomcat comet support during the last days. I've
>> > created
>> > a test application that simulates data streaming to multiple clients.
>> > Clients open a connection to the server and the connection is hold
>> > open. In
>> > the server there are several "sender" threads that leverage comet
>> > support to
>> > periodically send stream updates to the subscribed clients. Once a
>> client
>> > receives a specified amount of data the communication is closed.
>> >
>> > So, I'm running the test with tomcat svn version from 1 week ago. I 
>> have
>> > configured 150 tomcat threads, plus 25 acceptor threads. Tomcat 
>> runs in
>> a
>> > Sun-Fire-V240 box with 2 CPU cores. I have 75 sender threads that
>> > periodically sleep (100ms) to try to reduce the CPU usage. Once a 
>> sender
>> > thread wakes up, it will send all the data to the assigned clients.
>> >
>> > So basically the story is that the box is able to manage about 3500
>> > persistent comet connections smoothly. But when I raise the level to
>> 4000
>> > (80% CPU usage, 1.2G mem.) I start to get these exceptions for some 
>> (not
>> > all) connections:
>> >
>> > Exception in thread "MessageSender[]" java.lang.NullPointerException
>> >        at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
>> > InternalNioOutputBuffer.java:607)
>> >        at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
>> > InternalNioOutputBuffer.java:600)
>> >        at org.apache.coyote.http11.Http11NioProcessor.action(
>> > Http11NioProcessor.java:1032)
>> >        at org.apache.coyote.Response.action(Response.java:181)
>> >        at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
>> > InternalNioOutputBuffer.java:571)
>> >        at org.apache.coyote.Response.doWrite(Response.java:560)
>> >        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
>> > OutputBuffer.java:353)
>> >        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(
>> ByteChunk.java
>> > :434)
>> >        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java
>> :349)
>> >        at org.apache.catalina.connector.OutputBuffer.writeBytes(
>> > OutputBuffer.java:381)
>> >        at org.apache.catalina.connector.OutputBuffer.write(
>> > OutputBuffer.java:370)
>> >        at org.apache.catalina.connector.CoyoteOutputStream.write(
>> > CoyoteOutputStream.java:89)
>> >        at org.apache.catalina.connector.CoyoteOutputStream.write(
>> > CoyoteOutputStream.java:83)
>> >        at com.cognotec.streaming.CometRateSender.run(
>> CometRateSender.java
>> > :86)
>> >        at java.lang.Thread.run(Thread.java:595)
>> >
>> > So well, does anybody knows what can be the problem? Any suggestions?
>> > It can
>> > be also my problem, so I will be happy to provide more information.
>> >
>> > Cheers,
>> > Martin
>> >
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message