tomcat-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Thomas <>
Subject Re: Bug that spans tomcat and tomcat-native
Date Fri, 24 Jun 2016 10:08:10 GMT
On 23/06/2016 22:31, Nate Clark wrote:
> I tried to submit the bug but it seems that I am now unable to access
> Since you made it seem like it was important for this
> to be known about here is the info and patches.

Thanks. I'll chat with the infra folks and see if I can get whatever is
going on with BZ for you fixed.

> When performing some bench marking I noticed that the SSL performance
> of large request reads degraded heavily when performed after millions
> of small requests. Basically the setup is in a multi-threaded
> environment, about 200 threads, performing PUT requests using SSL with
> a body of about 4KB and then using 20 threads performing PUT requests
> with a body of 100MB. If the small requests are not performed the
> speed of the large requests in MB/s is about 2x.

Ouch. That is significant.

> I tracked down the issue to ERR_clear_err() blocking on an internal
> lock which protects a hash map of the error states. It seems that the
> hash map was growing unbounded because the states in it were never
> being cleared by a thread when it had completed SSL operations.
> According to OpenSSL documents ERR_remove_thread_state() or
> ERR_remove_state() for versions of OpenSSL less than 1.1.0 needs to be
> invoked prior to a thread exiting. This is not done by threads in the
> native code so the hash table keeps growing and getting larger and
> larger and more expensive to maintain.
> By adding a new native call which invoked ERR_remove_thread_state and
> calling it from AprEndpoint in tomcat I was able to reduce the
> contention on the lock and the performance improved.
> Because of the thread pool I could not find a simple clean way to
> invoke the cleanup before the thread dies so instead I added it to the
> end of the socket processing.
> Here are the patches I used against tomcat-native 1.1.34 and tomcat70:


I'm going to start some local performance testing to confirm I see
similar results and, assuming I do, I'll start looking at fixing this
for 1.2.x/9.0.x and back-porting.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message