tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob DeRemer <bob.dere...@thingworx.com>
Subject high CPU usage with NIO and 1000+ websockets on Tomcat 7.0.49?
Date Fri, 13 Dec 2013 13:12:08 GMT
Hi Mark (et al),

We're seeing really high CPU utilization with just a 1000 active websockets that are sending/receiving
100 byte binary messages.   The profiling snippet below was with 1000 connections.  If we
start to increase the number of connections, the time spent in the NIO logic starts to consume
the majority of the processing.  As a result, I'm trying to verify:


*         Is this something that can be tuned/improved using Tomcat's NIO settings?

*         Is this a bug in Tomcat 7 (websocket/NIO)?

*         Are we hitting some issue with 1000+ highly active NIO sockets?

The nature of this application is that it's primary functionality is to process binary messages
on (ideally 10 - 20K websockets), forward them to another server for processing, then send
the responses back.  As a result, the performance of the websocket communication is a critical
factor for success.

Please let me know if you'd like to discuss this - I'm available any time.

Thanks,
Bob


We have been profiling this in YourKit and as we increase the number of websockets, naturally
we see the CPU utilization increase.  The concern is that we're seeing this at what seems
to be a fairly low number of websockets.  The YourKit profiling showed 41% of the time being
spent in the following code.  My apologies for the formatting.  I can provide an attachment
out-of-band if it would be helpful.

|      +---org.apache.tomcat.util.net.NioEndpoint$Poller.run()                           
                        |  38,594   41 %  |              0  |
|      | |                                                                               
                        |                 |                 |
|      | +---sun.nio.ch.SelectorImpl.select(long)                                        
                        |  25,615   27 %  |              0  |
|      | | |                                                                             
                        |                 |                 |
|      | | +---sun.nio.ch.SelectorImpl.lockAndDoSelect(long)                             
                        |  25,615   27 %  |              0  |
|      | |   |                                                                           
                        |                 |                 |
|      | |   +---sun.nio.ch.WindowsSelectorImpl.doSelect(long)                           
                        |  25,615   27 %  |             46  |
|      | |     |                                                                         
                        |                 |                 |
|      | |     +---sun.nio.ch.WindowsSelectorImpl.resetWakeupSocket()                    
                        |  13,681   14 %  |          1,107  |
|      | |     |                                                                         
                        |                 |                 |
|      | |     +---sun.nio.ch.WindowsSelectorImpl$StartLock.access$300(WindowsSelectorImpl$StartLock)
            |  10,826   11 %  |              0  |
|      | |     |                                                                         
                        |                 |                 |
|      | |     +---sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys()                   
                        |   1,029    1 %  |              0  |
|      | |     |                                                                         
                        |                 |                 |
|      | |     +---sun.nio.ch.SelectorImpl.processDeregisterQueue()                      
                        |      31    0 %  |              0  |
|      | |                                                                               
                        |                 |                 |
|      | +---org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(SelectionKey, NioEndpoint$KeyAttachment)
   |  11,996   13 %  |             31  |
|      | | |                                                                             
                        |                 |                 |
|      | | +---org.apache.tomcat.util.net.NioEndpoint.processSocket(NioChannel, SocketStatus,
boolean)            |  11,778   12 %  |              0  |
|      | | | |                                                                           
                        |                 |                 |
|      | | | +---org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(Runnable)     
                        |  11,746   12 %  |              0  |
|      | | | | |                                                                         
                        |                 |                 |
|      | | | | +---org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(Runnable, long,
TimeUnit)            |  11,746   12 %  |              0  |
|      | | | |   |                                                                       
                        |                 |                 |
|      | | | |   +---java.util.concurrent.ThreadPoolExecutor.execute(Runnable)           
                        |  11,746   12 %  |             15  |
|      | | | |     |                                                                     
                        |                 |                 |
|      | | | |     +---org.apache.tomcat.util.threads.TaskQueue.offer(Object)            
                        |  11,731   12 %  |              0  |
|      | | | |       |                                                                   
                        |                 |                 |
|      | | | |       +---org.apache.tomcat.util.threads.TaskQueue.offer(Runnable)        
                        |  11,731   12 %  |             46  |
|      | | | |         |                                                                 
                        |                 |                 |
|      | | | |         +---java.util.concurrent.LinkedBlockingQueue.offer(Object)        
                        |  11,668   12 %  |             15  |









Bob DeRemer
Senior Director, Architecture and Development

[Description: Description: Description: Description: cid:image001.png@01CBE3DE.51A12030]
http://www.thingworx.com<http://www.thingworx.com/>
Skype: bob.deremer.thingworx
O: 610.594.6200 x812
M: 717.881.3986


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message