Mailing-List: contact dev-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Developers List" <dev@tomcat.apache.org>
Subject: Re: Bug that spans tomcat and tomcat-native
To: Tomcat Developers List <dev@tomcat.apache.org>
References: <CAG2YfWPAK9OL50tXB5H987bexmb8BZ2TvX9PddHfo=q26HAvXA@mail.gmail.com>
 <2e6f56c1-32cb-28c6-3864-7425aa973069@apache.org>
 <CAG2YfWPMieaN2far8Pzti-pYfKoVkQc9MG0_aXEtMJ-KaL6+XQ@mail.gmail.com>
From: Mark Thomas <markt@apache.org>
Message-ID: <7afb790c-67dc-7dd5-56a0-78961479f571@apache.org>
Date: Fri, 24 Jun 2016 11:08:10 +0100
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.1.1
MIME-Version: 1.0
In-Reply-To: <CAG2YfWPMieaN2far8Pzti-pYfKoVkQc9MG0_aXEtMJ-KaL6+XQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 24 Jun 2016 10:08:22 -0000

On 23/06/2016 22:31, Nate Clark wrote:
> I tried to submit the bug but it seems that I am now unable to access
> bz.apache.org. Since you made it seem like it was important for this
> to be known about here is the info and patches.

Thanks. I'll chat with the infra folks and see if I can get whatever is
going on with BZ for you fixed.

> When performing some bench marking I noticed that the SSL performance
> of large request reads degraded heavily when performed after millions
> of small requests. Basically the setup is in a multi-threaded
> environment, about 200 threads, performing PUT requests using SSL with
> a body of about 4KB and then using 20 threads performing PUT requests
> with a body of 100MB. If the small requests are not performed the
> speed of the large requests in MB/s is about 2x.

Ouch. That is significant.

> I tracked down the issue to ERR_clear_err() blocking on an internal
> lock which protects a hash map of the error states. It seems that the
> hash map was growing unbounded because the states in it were never
> being cleared by a thread when it had completed SSL operations.
> 
> According to OpenSSL documents ERR_remove_thread_state() or
> ERR_remove_state() for versions of OpenSSL less than 1.1.0 needs to be
> invoked prior to a thread exiting. This is not done by threads in the
> native code so the hash table keeps growing and getting larger and
> larger and more expensive to maintain.
> 
> By adding a new native call which invoked ERR_remove_thread_state and
> calling it from AprEndpoint in tomcat I was able to reduce the
> contention on the lock and the performance improved.
> 
> Because of the thread pool I could not find a simple clean way to
> invoke the cleanup before the thread dies so instead I added it to the
> end of the socket processing.
> 
> Here are the patches I used against tomcat-native 1.1.34 and tomcat70:

Thanks.

I'm going to start some local performance testing to confirm I see
similar results and, assuming I do, I'll start looking at fixing this
for 1.2.x/9.0.x and back-porting.

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org