tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Perez" <mper...@gmail.com>
Subject Re: Exception with Comet in Tomcat with high load
Date Wed, 02 May 2007 09:54:32 GMT
Sebastiaan,

Yes you're right. I forgot to add something like isClosed() to your
SynchronizedOutputStream so I can check if it's closed before writing.

Anyways, I found that after a certain limit (for ex. 6000 -> 8000 users) the
system starts to behave incorrectly. Some clients never get an answer from
the server, some clients only receive partial data, and some of them really
end. I guess I found the limit in the box, but attached are the source code
files if someone wants to see the sample or simply spot something in it.

Cheers,
Martin

On 5/1/07, Sebastiaan van Erk <sebster@sebster.com> wrote:
>
> Hi,
>
> Did you synchronize the event.close() with same lock you use to write to
> the output stream (the SynchronizedOuputStream instance)?
>
> I saw exactly the same stack trace you are getting due to (incorrect)
> concurrent access to the CoyoteOutputStream. The NullPointerException
> suggest that the request has already been cleaned up/recycled and yet
> you still write to it. Make sure that when you close the event, either
> the write becomes a null operation, or better, does not get called. It
> is a bug to call write on the response output stream after the event has
> been closed.
>
> With the SynchronizedOutputStream construction I am relying on Tomcat
> not to close or call any methods on the CoyoteOutputStream outside of
> the CometProcessor's event() method. I don't know if this is correct (it
> has worked for me in practice so far, and I've been doing some heavy
> duty testing); however I do not know what else I could synchronize on.
> If anybody knows a better approach, I would like to know. :-)
>
> Regards,
> Sebastiaan
>
> Martin Perez wrote:
> > Hi,
> >
> > Thanks for the tip Sebastian, but it doesn't work for me. I added your
> > class, wrapped all the output streams and synchronized close events
> > but the
> > result is the same.
> >
> > Exception in thread "MessageSender[]" java.lang.NullPointerException
> >        at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
> > InternalNioOutputBuffer.java:607)
> >        at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
> > InternalNioOutputBuffer.java:600)
> >        at org.apache.coyote.http11.Http11NioProcessor.action(
> > Http11NioProcessor.java:1032)
> >        at org.apache.coyote.Response.action(Response.java:181)
> >        at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
> > InternalNioOutputBuffer.java:571)
> >        at org.apache.coyote.Response.doWrite(Response.java:560)
> >        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
> > OutputBuffer.java:353)
> >        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(
> ByteChunk.java
> > :434)
> >        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java
> :349)
> >        at org.apache.catalina.connector.OutputBuffer.writeBytes(
> > OutputBuffer.java:381)
> >        at org.apache.catalina.connector.OutputBuffer.write(
> > OutputBuffer.java:370)
> >        at org.apache.catalina.connector.CoyoteOutputStream.write(
> > CoyoteOutputStream.java:89)
> >        at org.apache.catalina.connector.CoyoteOutputStream.write(
> > CoyoteOutputStream.java:83)
> >        at com.cognotec.streaming.SynchronizedOutputStream.write(
> > SynchronizedOutputStream.java:25)
> >        at com.cognotec.streaming.CometRateSender.run(
> CometRateSender.java
> > :86)
> >        at java.lang.Thread.run(Thread.java:595)
> >
> > (As I said, I also synchronized all the close method calls either in
> > stream
> > either in comet event)
> >
> > So, the question is: Is this a bug or simply I have reached the
> > scalability
> > limit for my box?
> >
> > Cheers,
> > Martin
> >
> > On 4/30/07, Sebastiaan van Erk <sebster@sebster.com> wrote:
> >>
> >> Hi,
> >>
> >> I had the same problems. It was a synchronization issue with me. What I
> >> did was make a synchronized output stream wrapper (see attached file)
> >> which wraps all access to the output stream, and I synchronized on this
> >> output stream whenever I called event.close() on the Comet event
> >> (especially in the READ event). When doing serveral IO operations on
> the
> >> output stream which should not be interrupted, you can simply
> >> synchronize on it (since SynchronizedOutputStream locks on itself).
> >>
> >> Regards,
> >> Sebastiaan
> >>
> >>
> >> Martin Perez wrote:
> >> > Hi all,
> >> >
> >> > I have been testing Tomcat comet support during the last days. I've
> >> > created
> >> > a test application that simulates data streaming to multiple clients.
> >> > Clients open a connection to the server and the connection is hold
> >> > open. In
> >> > the server there are several "sender" threads that leverage comet
> >> > support to
> >> > periodically send stream updates to the subscribed clients. Once a
> >> client
> >> > receives a specified amount of data the communication is closed.
> >> >
> >> > So, I'm running the test with tomcat svn version from 1 week ago. I
> >> have
> >> > configured 150 tomcat threads, plus 25 acceptor threads. Tomcat
> >> runs in
> >> a
> >> > Sun-Fire-V240 box with 2 CPU cores. I have 75 sender threads that
> >> > periodically sleep (100ms) to try to reduce the CPU usage. Once a
> >> sender
> >> > thread wakes up, it will send all the data to the assigned clients.
> >> >
> >> > So basically the story is that the box is able to manage about 3500
> >> > persistent comet connections smoothly. But when I raise the level to
> >> 4000
> >> > (80% CPU usage, 1.2G mem.) I start to get these exceptions for some
> >> (not
> >> > all) connections:
> >> >
> >> > Exception in thread "MessageSender[]" java.lang.NullPointerException
> >> >        at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(
> >> > InternalNioOutputBuffer.java:607)
> >> >        at org.apache.coyote.http11.InternalNioOutputBuffer.commit(
> >> > InternalNioOutputBuffer.java:600)
> >> >        at org.apache.coyote.http11.Http11NioProcessor.action(
> >> > Http11NioProcessor.java:1032)
> >> >        at org.apache.coyote.Response.action(Response.java:181)
> >> >        at org.apache.coyote.http11.InternalNioOutputBuffer.doWrite(
> >> > InternalNioOutputBuffer.java:571)
> >> >        at org.apache.coyote.Response.doWrite(Response.java:560)
> >> >        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(
> >> > OutputBuffer.java:353)
> >> >        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(
> >> ByteChunk.java
> >> > :434)
> >> >        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java
> >> :349)
> >> >        at org.apache.catalina.connector.OutputBuffer.writeBytes(
> >> > OutputBuffer.java:381)
> >> >        at org.apache.catalina.connector.OutputBuffer.write(
> >> > OutputBuffer.java:370)
> >> >        at org.apache.catalina.connector.CoyoteOutputStream.write(
> >> > CoyoteOutputStream.java:89)
> >> >        at org.apache.catalina.connector.CoyoteOutputStream.write(
> >> > CoyoteOutputStream.java:83)
> >> >        at com.cognotec.streaming.CometRateSender.run(
> >> CometRateSender.java
> >> > :86)
> >> >        at java.lang.Thread.run(Thread.java:595)
> >> >
> >> > So well, does anybody knows what can be the problem? Any suggestions?
> >> > It can
> >> > be also my problem, so I will be happy to provide more information.
> >> >
> >> > Cheers,
> >> > Martin
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To start a new topic, e-mail: users@tomcat.apache.org
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
View raw message