From users-return-268763-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Mon Nov 18 12:01:06 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7FDA6180657 for ; Mon, 18 Nov 2019 13:01:05 +0100 (CET) Received: (qmail 42258 invoked by uid 500); 18 Nov 2019 12:01:01 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 42248 invoked by uid 99); 18 Nov 2019 12:01:01 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.42) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Nov 2019 12:01:01 +0000 Received: from [192.168.23.12] (host213-121-9-136.range213-121.btcentralplus.com [213.121.9.136]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id B5EAA1004 for ; Mon, 18 Nov 2019 12:01:00 +0000 (UTC) Subject: Re: tomcat thread incurring CPU load To: users@tomcat.apache.org References: <0F7034A9-A36D-4782-A3EC-48F58F4A5B72@apache.org> <57d65705-9eef-5644-4e12-a949eec57871@apache.org> <2b5ae4f0-95d3-33f9-121c-cdb4690ab959@christopherschultz.net> From: Mark Thomas Openpgp: preference=signencrypt Autocrypt: addr=markt@apache.org; prefer-encrypt=mutual; keydata= mQINBEq0DukBEAD4jovHOPJDxoD+JnO1Go2kiwpgRULasGlrVKuSUdP6wzcaqWmXpqtOJKKw W2MQFQLmg7nQ9RjJwy3QCbKNDJQA/bwbQT1F7WzTCz2S6vxC4zxKck4t6RZBq2dJsYKF0CEh 6ZfY4dmKvhq+3istSoFRdHYoOPGWZpuRDqfZPdGm/m335/6KGH59oysn1NE7a2a+kZzjBSEg v23+l4Z1Rg7+fpz1JcdHSdC2Z+ZRxML25eVatRVz4yvDOZItqDURP24zWOodxgboldV6Y88C 3v/7KRR+1vklzkuA2FqF8Q4r/2f0su7MUVviQcy29y/RlLSDTTYoVlCZ1ni14qFU7Hpw43KJ tgXmcUwq31T1+SlXdYjNJ1aFkUi8BjCHDcSgE/IReKUanjHzm4XSymKDTeqqzidi4k6PDD4j yHb8k8vxi6qT6Udnlcfo5NBkkUT1TauhEy8ktHhbl9k60BvvMBP9l6cURiJg1WS77egI4P/8 2oPbzzFiGFqXyJKULVgxtdQ3JikCpodp3f1fh6PlYZwkW4xCJLJucJ5MiQp07HAkMVW5w+k8 Xvuk4i5quh3N+2kzKHOOiQCDmN0sz0XjOE+7XBvM1lvz3+UarLfgSVmW8aheLd7eaIl5ItBk 8844ZJ60LrQ+JiIqvqJemxyIM6epoZvY5a3ZshZpcLilC5hW8QARAQABtCJNYXJrIEUgRCBU aG9tYXMgPG1hcmt0QGFwYWNoZS5vcmc+iQI3BBMBCgAhBQJKtA7pAhsDBQsJCAcDBRUKCQgL BRYCAwEAAh4BAheAAAoJEBDAHFovYFnn2YgQAKN6FLG/I1Ij3PUlC/XNlhasQxPeE3w2Ovtt weOQPYkblJ9nHtGH5pNqG2/qoGShlpI04jJy9GxWKOo7NV4v7M0mbVlCXVgjdlvMFWdL7lno cggwJAFejQcYlVtxyhu4m50LBvBunEhxCbQcKnnWmkB7Ocm0Ictaqjc9rCc1F/aNhVMUpJ0z G1kyTp9hxvN6TbCQlacMx5ocTWzL0zn6QZhbUfrYwfxYJmSnkVYZOYzXIXIsLN5sJ9Q4P8tj Y4qWgd+bQvOqPWrkzL9LVRnGOrSYIsoM5zWdoj1g1glMzK/ZqJdRqqqBhe6FYTbXipz8oX8i mCebcaxZnfLhGiqqX+yDa3YUwDiqom+sZOc0iXGvKkqltPLpNeF0MVT7aZjalsQ/v2Ysb24R Ql9FfjfWmvT8ZPWz8Kore1AI4UcIIgFVtM+zuLlL9CIsGjg+gHDE2dhZDY0qfizlHL9CoAWU DM3pIfxM2V4BRn1xO+j/mModhjmYLZvnFVz4KGkNO7wRkofAANIWYo3WI5x83BGDH371t3NR rrpSSFP0XpQX6/Leaj2j6U6puABL2qBxhscsO6chc3u4/+019ff+peZVsc9ttcTQXsKIujmM b8p2sk5usmv6PKVX3oW/RAxpbVHU5kZ5px1Hq7mMQdZfLs5ff4YymXBH02z4/RmSzPam0Xb5 uQINBEq0DukBEADCNEkws5YroBmbu8789Xf006gTl5LzD/Hdt3sAp9iCfPgucO+l7U+xbo1X HTMJQwEVfS+Rx3RbaLYRG+hU7FuJLQB/5NaCDNRuqw5KHyQtJUH+zo84IqqfMzG8aOSdHg1y r2xKH4QTmgQONBu/W0xEZmZro6TjYNwkk2pwXK2yuImZPUOy+mK1qF8Wm3hTtkPE+FFSNFIa eHDoTGmx/0Riu/K7dNJTrC0TlRpn2K6d60zB53YYTc+0DYSDyB0FupXiAx/+XEGn3Q7eNi2B V6w50v5r51QP8zptiFflMfFKNAfV8xS5MteQd98YS5qqd/LPo3gS5HFPQaSL0k3RTClv7fQN HcZFqmv0OWpix6zm2npYxhqsTDGeSa52/uXehVXF5JubYFifMSLpbGVZqdrmG5hr2cycxsjF iY0zJOaRitmN/JWbOGLiwrcN4ukKNyFntFG5jPaFnJdx9rHfyJNeF9cgv9JlZeFxJ6WqIAhl KOuH3K8/py0SPE6ZOFfRo0YUxvh25K/siOcPLm613aOxyY7YfQ8ME2vgn7I0mAtg9am+YFDa bGqj839odwZdzZv2T2mUHnybFTJFBuMWGWKYstYDS6eZEmhupbPvUKkDug/mO+gdo+pSKF9Y S6DM5RtCdTNJq4NZY50ypBb5RSj+INHPocIp2V/DDTbzySsu6wARAQABiQIfBBgBCgAJBQJK tA7pAhsMAAoJEBDAHFovYFnnLe0P/i34oK5cE2LlqUEITEcTO94x1EX0UmtKokRfQ3AYWK8X eFD8cmSty72hMkL+1c0V//4Qc53SUyLIWXk8FKWF7hdL3zyuBqlRb55721CYC35GA/jR90p0 k1vr701gaat2cNTOVC0/6H9cE5yYXT+zMr9TSiKCDwONhhSbmAJZc6X0fgsmCD7I5xUI5Vri hN/Wx0CZBtrXGUyE4hgFaYSGptZmkY5Ln1e+nI185Bda7bpLwcAIGrI9nYtVXgf71ybGKdPP tFfXIoPXuctn99M7NnWBhNuGDms2YWkOC7eeWBTxKkZDWR3vRmRy52B6GxR7USk/KXs7yqGP kfT/c4CZFfOurZUXXuC3PvOme0DQmqwExtJormoG4Fy6suEFPrfhYMigTy7kSbVTCOBMjQLH +U/FFNshvg9+M/ZvaKT+0lpRvBSuG5ngsC0bO0xWsXhb6qfH2h53g4VcwFvCBL5IfqgAeUbC nGGHNcGWpmwdeb7D7ahrNZSHEUUYR7lTbjkYS01/QDOcEwNZOqDRIJUQOOUq35721VeROkdh ZmMZtFlsQeQJsWoqGrQo/kEYicVlMVOgjmOOzOa5fRb/IqlGlBn4a4me3hWthLLtMy+OOEim 6ENjntVTBQiTP/YqrxWDbCkaD7b2e9wY5N3JlRxMIQHfcHaND3PRdQSn7oHYXmJl Message-ID: <9ac5a1a9-025e-f9e3-59a5-62ec9bcf142b@apache.org> Date: Mon, 18 Nov 2019 12:00:59 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <2b5ae4f0-95d3-33f9-121c-cdb4690ab959@christopherschultz.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 8bit OK, it looks like I can reproduce this. Steps to reproduce: - Windows 2016 Server fully patched - Java 1.8.0u144 - Install Tomcat 8.5.45 from windows installer - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 - Modify server.xml to use Http11AprProtocol on port 8080 - Make a single request I then see 1 core running at 100% until the connection times out after 20s. Make another request and a core goes back up to 100% for 20s (the default keep-alive time out). Next steps are to try and track down the root cause. Mark > Mark and M, > > On 11/13/19 19:31, Mark Thomas wrote: >> On November 13, 2019 11:42:34 PM UTC, "M. Manna" >> wrote: >>> I see this update on Windows which may have been responsible >>> (suspicion only, haven’t rolled it back yet) >>> >>> >>> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr > ocode-updates >>> >>> >>> > Was 8.5.45 built on Windows 10 in presence of this update ? > >> No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully >> patched at the time of the build Windows 7 64-bit VM. > Also it doesn't matter because binaries don't include CPU microcode. > > It's more likely that the target system has microcode updates such as > these that may negatively impact performance. > > -chris > >>> >>> Thanks, >>> >>> On Wed, 13 Nov 2019 at 17:55, M. Manna >>> wrote: >>> >>>> Hi Chris, >>>> >>>> On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < >>>> chris@christopherschultz.net> wrote: >>>> >> On 11/13/19 11:20, M. Manna wrote: >>>>>>> HI Mark, >>>>>>> >>>>>>> On Wed, 13 Nov 2019 at 15:38, Mark Thomas >>>>>>> wrote: >>>>>>> >>>>>>>> On 12/11/2019 19:11, M. Manna wrote: >>>>>>>>> HI Mark, >>>>>>>>> >>>>>>>>> following my previous reply, we have now confirmed >>>>>>>>> that it's indeed >>>>>>>> 8.5.45 >>>>>>>>> with APR 1.2.23 that's causing such high JVM CPU >>>>>>>>> usage. We used took out 2 out of 50 servers from the >>>>>>>>> load balancer config, reverted tomcat, and >>>>>>>>> redeployed. With near to identical user traffic, the >>>>>>>>> two servers are responding normally without/without >>>>>>>>> traffic with 8.5.41. The JVM dump looks a lot better >>>>>>>>> with 8.5.41. >>>>>>>>> >>>>>>>>> We do think that the recent changes in APR and some >>>>>>>>> other tomcat jar may have caused compatibility issue >>>>>>>>> on Windows server 2016 (64-bit) platform. But >>>>>>>>> unfortunately, we cannot pinpoint exactly what change >>>>>>>>> may have caused this (i.e. actual OS vs Security >>>>>>>>> Updates). With this in mind, we are also being wary >>>>>>>>> to move to 8.5.47 as we don't know if the same issue >>>>>>>>> will >>>>>>>> occur >>>>>>>>> again. Since 8.5.41 has been packaged with previously >>>>>>>>> accepted >>>>>>>> application >>>>>>>>> installer, we are more comfortable rolling back. >>>>>>>> >>>>>>>> Just to confirm, you see this high CPU usage with a >>>>>>>> clean install (no additional web applications deployed, >>>>>>>> no configuration changes) on Windows 2016 DataCenter >>>>>>>> (64-bit)? >>>>>>>> >>>>>>>> If this is the case, it should be fairly easy to >>>>>>>> reproduce. >>>>>>>> >>>>>>>> Mark >>>>>>>> >>>>>>>> We do not deploy multiple applications. In fact, Under >>>>>>>> tomcat >>>>>>> webapps/ROOT we only have one application (ours). Each >>>>>>> tomcat instance is hosted on a VM (total 50) and all of >>>>>>> them are identically configured (server.xml, web.xml, >>>>>>> logging, CPU/RAM). We have not made any other >>>>>>> configuration change between 8.5.41 and 8.5.45. And yes, >>>>>>> I agree with you that it's fairly easy to reproduce. > >> I think the question is whether or not your application is >> required >>>> to >> be deployed. Can you reproduce this issue with just the stock >> applications bundled with Tomcat? > >>>>> >>>>> My apologies, but our application needs to be deployed. We >>>>> have not >>>> (or >>>>> didn't try in the past) to simply deploy tomcat with stock >>>> application (in >>>>> other words, simply starting the tomcat OOB) on our prod >>>>> servers. This is the first time it has hit us with such >>>>> disparity. I’ll try to investigate and get a stock >>>>> application data. But we may not be able >>>> to do >>>>> that quite easily as it’s in our production. >>>>> >>>>> What I can see is that 3 Windows updates may have been >>>>> responsible >>>> for >>>>> this, but we aren’t sure about that. I’ll let you know if we >>>>> can get anything with the stock application instance. >>>>> >>>>> Thanks, >>>>> >>>>> - -chris >>>>> >>>>> >>>>> >>> --------------------------------------------------------------------- >>>>> >>> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>> For additional commands, e-mail: >>>>> users-help@tomcat.apache.org >>>>> >>>>> > > >> --------------------------------------------------------------------- > > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >> For additional commands, e-mail: users-help@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org > For additional commands, e-mail: users-help@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org