tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob DeRemer <bob.dere...@thingworx.com>
Subject seeing RemoteEndpoint.Basic BINARY_FULL_WRITING errors under load in Tomcat 7.0.48-dev (trunk)
Date Wed, 27 Nov 2013 17:28:23 GMT
When testing our WebSocket application under heavy load we eventually get the following exception
from the JSR-356 ServerEndpoint's RemoteEndpoint.Basic when trying to write our binary response
message:

java.lang.IllegalStateException: The remote endpoint was in state [BINARY_FULL_WRITING] which
is an invalid state for called method
       at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.checkState(WsRemoteEndpointImplBase.java:1048)
~[tomcat7-websocket.jar:7.0.48-dev]
       at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.binaryStart(WsRemoteEndpointImplBase.java:1001)
~[tomcat7-websocket.jar:7.0.48-dev]
       at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendBytes(WsRemoteEndpointImplBase.java:118)
~[tomcat7-websocket.jar:7.0.48-dev]
       at org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendBinary(WsRemoteEndpointBasic.java:43)
~[tomcat7-websocket.jar:7.0.48-dev]
       at com.thingworx.communications.server.connection.Jsr356ServerConnection.sendBinaryMessage(Jsr356ServerConnection.java:240)
~[thingworx-server-common.jar:na]
       at com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendBinaryMessage(CommunicationEndpoint.java:321)
~[thingworx-server-common.jar:na]
       at com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendMessage(CommunicationEndpoint.java:303)
~[thingworx-server-common.jar:na]
       at com.thingworx.apiserver.endpoints.RoutingServerEndpoint.sendMessage(RoutingServerEndpoint.java:24)
~[thingworx-apiserver.jar:na]
       at com.thingworx.apiserver.routing.OutboundMessageRoutingTask.run(OutboundMessageRoutingTask.java:79)
~[thingworx-apiserver.jar:na]
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
       at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

The Tomcat build is 7.0.48-dev (from today, 11/27) and we're running on Ubuntu Server 13.10
in EC2 (16 vCPUs).  The client created 1000 concurrent websockets, and is using 100 threads
to send 100 concurrent requests every 3 ms.  The server receives the messages, processes them,
then sends the response back to the appropriate client.  Many response are sent successfully,
but after some period of time, we start receiving the above errors when attempting to send
the responses back.

Is it possible that we could be filling up the TCP buffers (or something related) and this
would cause the above exception?

Please note: we lock around a specific ServerEndpoint's RemoteEndpoint.Basic - to make sure
we aren't trying to send more than 1 message at a time.

Please let me know if this is a bug, or if you need additional information.

Thanks,
Bob



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message