From users-return-268746-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Wed Nov 13 16:27:03 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 835B0180621 for ; Wed, 13 Nov 2019 17:27:03 +0100 (CET) Received: (qmail 48377 invoked by uid 500); 13 Nov 2019 16:26:59 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 48365 invoked by uid 99); 13 Nov 2019 16:26:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Nov 2019 16:26:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D899B1812DE for ; Wed, 13 Nov 2019 16:26:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=christopherschultz-net.20150623.gappssmtp.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id FtYnsH9pYUNR for ; Wed, 13 Nov 2019 16:26:57 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.219.179; helo=mail-yb1-f179.google.com; envelope-from=chris@christopherschultz.net; receiver= Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 5F582BC509 for ; Wed, 13 Nov 2019 16:26:57 +0000 (UTC) Received: by mail-yb1-f179.google.com with SMTP id i7so1153734ybk.12 for ; Wed, 13 Nov 2019 08:26:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=christopherschultz-net.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:openpgp:autocrypt:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=wWp4ylD/39VupXQfvVGsSIXXkjmAakkObatcG8MFu5U=; b=GyUhAAxZN+iRj8yjmut76dTKP/9YB1Q98gsX5YgpJ7WxN3WFqZ3hN/2XmQSGERVvFI 67GKb28iV/DYHUKWiGQZ8mUh1+g8JmqHhYYCz8CFOIpgNvLqsZ7zXBltVOytYuaD1iyd UfOb3tzy1QpNP2J4axDT9C0cn3fNFooiXdSbkkvVu9HPHN3SsdO3QLTvLznz/JkThAhL j9c1oI6gkRWPYtt7nlCRpH4f3Fd5fFnU9YvdJlZbEvzXHAAiEfM0KvLugEHYKfTEFUoH X5UVl0qMRnDF0j9WfEe8mfabDT9I7k2eor7e/9DvYVeICvJcvh5e+cSCLhsvggNMs3l0 AAfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=wWp4ylD/39VupXQfvVGsSIXXkjmAakkObatcG8MFu5U=; b=rn4hg/2TOcyV+YZvF7uehhq9uAWQgWNKncremwkaB45XlU3VmrwAsm1qfNqo9Vr0Z+ hKb/8LFOzHp7RvjK/LtXNnzrm2r67yMr5kAsDe3Pz5N1LDWR6gS+kj3NcsRh2HL30lRx m0QQLTf2x/ZKMaxpBUFKOBJrReSIxoitiHuSbpsPbk21j6vWDz3fYWx3ueehOXVfzxMM /i9N2G56d6gMCKjPT+fvWZsboxfpJyHT7asSZCrYfKUvpfIydu8hnCTQK70Zo7NNERWM ZLV1esBa+zHuVQffDDXneVihiioDZqX+lZ8u6qcMbsY/COYDOyCmZ4M9V4xVJ7YZxlpb ueRg== X-Gm-Message-State: APjAAAV3hwZ0i2iickfxuYMPfxZD1xxPNQBTzgt2MaRwQO7+9A892wh3 c4nkRCvawrYG0PnNZmgv6W6yTh2z/rY= X-Google-Smtp-Source: APXvYqyzAoBW9rPPGsJVqQo8sbs3M8+K7U9NfqfOQ+BTtCUM2PTEtcf/lj46hwWJ65HJH9S+mnLssA== X-Received: by 2002:a25:19c5:: with SMTP id 188mr3574833ybz.253.1573662410195; Wed, 13 Nov 2019 08:26:50 -0800 (PST) Received: from Christophers-MacBook-Pro-2.local (pool-108-48-175-111.washdc.fios.verizon.net. [108.48.175.111]) by smtp.gmail.com with ESMTPSA id j188sm1127164ywd.50.2019.11.13.08.26.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Nov 2019 08:26:49 -0800 (PST) Subject: Re: tomcat thread incurring CPU load To: users@tomcat.apache.org References: <0F7034A9-A36D-4782-A3EC-48F58F4A5B72@apache.org> <57d65705-9eef-5644-4e12-a949eec57871@apache.org> From: Christopher Schultz Openpgp: preference=signencrypt Autocrypt: addr=chris@christopherschultz.net; keydata= xsFNBE+pgz4BEADd7qAWgqXcNltlB3aow0UneRmNSVjHKgekgs0ZXxG9l50Athksr/3bL/yg bxFB00JcM9W+UxLhKHiMSyzfeBHn9l9wAlLFKs0S91KXTUnRwGFtvgstvGROoqPgTVREklnm yW/KpzOwqSrQ5xHcogaT+XWlXmRbtFypi52Z5HGWlFWWgwx0vKBWHmQayPtCif0v1RDxfdV9 zziodn0TnpfBQsEgf9TDAjkNT8f0ecwTnhSihTDm1W5HCK7Pm5DfUtree1Oh6Ncz2ljlUO0b 3Lai9pX48eZOj7WQXPefkcv2AoUvdELkQKw3klM5YNXbXPf1KAjky+q4DQ1ydD6LkK+9cI3S TeMesTlk/tytOsaN2NH2k87sEpcumbH0AcmPFEnIYUfm4KzWdKlYA6mbV3Pk3tHSuayyJovj h/7Y7BG9p2l7D60r49hzrTPG8VxNkSliNLcSjI3QjYpfhSlqmqXyVKzdzirK1HPr1xfJStig RpLP9nWarZjoXng9N0etGwtH/8roeDPYA8x9ba1KXy/1g/i+RLx2ms+rueCpnFZxU3GZNUSp RfpdUbwCN3Zm1w5Z6SI8X2aSnWWeYzU6HMsV+P4PROnFsgxDeOpyWhyEaaVLXQtOYwcHneHb n56vSG50TkAuHs5kk/3/YDPSsqjsUPOuhKgFMh3iqMTh5DMdSwARAQABzTJDaHJpc3RvcGhl ciBTY2h1bHR6IDxjaHJpc0BjaHJpc3RvcGhlcnNjaHVsdHoubmV0PsLBegQTAQgAJAIbLwUL CQgHAwUVCgkICwUWAgMBAAIeAQIXgAUCT6mETgIZAQAKCRDzrVyUpn9wflkxD/9IsahRqHTV /hH5nuPqVO692cQqHvPtMPO6lDb4909VN5T1i+1hFr80P0KVDL6EI78lDBJ2TThWI0o5vFdm sRlei59wsgTvkKTph5QwwOWl7OyzUDX3WbKhkNQdGf4I+/g/1s2bHaRoG30ELdL7cwUPCPrW 0KQwBy7Rtr0WbdujKOw9b/UcgyXEOE1wNcorq/E1o5/6BRYIcFQOO4sjHjGcChOpSg5ms4zb s+Xv3gOtLrbmOPRTXdvBxwJA6kkfQFHvI42kXYghTdqhBVPnHYPqUeavRsb+Yz3ghkZhj35i GfaGyXNwFBikCYjzIaj44NOkT1pU50MgIbjSJ+xoHnC20T942kekqp6wzqUM19Pa9ohsEdA1 Sf6/A7RmpZRrxSIY02ZVnGccnVjglnylVcnxrNAZC3ebxCeZPQ09FBR0Uqlsrdt7A3hlEP2F aoMTSa+hYqfWBGB7uZhcJZIsZspxm8J0txeOzYNSFDl7mF134ShRsq6dpSugCdcdeSWKliBz q0U8sIabOFLMxM0hbwkn2RG4OaurJLWXQf+7IhA/J8TizjkbdxLmR2PiTiVtrx484mpWpbF8 po/em0q/reFnL+JtOM6qlJE/Q4B6PfkchhU5vKPfmGw98t9guyw5G8YSR1rR+SOowHg4T/i2 Rezz1idKmoFpPdNFRPlOAC+d687BTQRPqYM+ARAAzEItVpzvcgZB+faUWi54lJoA8GnVxXEe OQY+7wk/P5i9GtL0UVXC53j2F87BDVXGalKgVjEVdNY3Cyx+dJ2os65gjxd6ZK18zc6N7YZB Z00XNU9nTz5XImZzHn4VmeXYMQrKO/981nCNPlV6CVdgGg9wl1Ij5Sh8SSTb8kWSo1ngx+XX 4yJNUbfSh32yMPVGI7ZcoZLm9gdgTOOnuEkeeGs/lPvYN+1Cv/YtvkPybSOSWSdHxIVU4Iko 6V7IkM1amjdwKfoeg+CLhZsbY7VLAzVtGvaF5z4rtJtCfTfhbYD0wS8afEBcvsew1HdtYDT5 AJqojeZBGDuY7JCgALc3HCy34Zzk+mi1qwvrm5i/CBMuIvjxB2MkzhHQNUD20fzdRcoIgw4J IzbqZLlOpVFehDXzKT/h5vh+Uv7s6Rz5gP5i0Rkcghw00mRBvuN8mpQnLt4hYL22cNh/tk0L Fxda7ZaPehu7ug4E5FEB0Ifm1KV18P7Kpfu8tiSLz7rl++x73o4uv4bk1ZnjO/jFsx0KLGwq VxR276ZIwsV4WpLYvJ5fR0kqqd/TOKXGSEA0eGxWTeb/fNtkYemRVoasB1+rqjh/Rz0p20o8 elkqDhpzzhrMNzEMYkLySu7npWCeWW4Nd6097+OG9BCLO+ndGmAcupdu6WMEj2UlWsQxuCYC PgsAEQEAAcLDfgQYAQgACQUCT6mDPgIbLgIpCRDzrVyUpn9wfsFdIAQZAQgABgUCT6mDPgAK CRAc8Ck/pTykWO6WD/0XlAG4D4GwzzuOfh7DG6cm/I0vmASEJkY5ghStW4GUbYosgS/btyj/ YPWzVh4HWMvuA6YYKCuz/CM3h34dR25XmHqUdOyJOCnMJ3psdv5YsytgnEdvINZALlDdBX3G sfytgS0KnVjAc92LfJOxHAsZf4zE3SU28FMX7jCgeqO3YrvkHsZ8dzzgw3QYT0J3NcYfkflb DPBXBDGrvdUuea/w6F17pctdRdt7jE3JiLFq2F9ehXOSsIwecUlqVYiCRuxblD4cJ6gKMn0y 8zllW4GyIbf/+mNLkpKoMPYnptDvcEojluHtwbkSfF5AwgJbm6pfs9a2vpGBVko+dBXGh4/T 3qNYxeGEAsI0psEJu3EZN9dYv/ZOb69DUJ6SwEKp/L7lU7C8HoLx/MpKtuJO9OS5uuAhdBSi GqfaN9zP2NxPXSwnexVK2exy/h5sUevDsnBEHmyxe5GRSrIilyijLtlYhq2W7G95poxIFZuL Db98R+7VR9Yl9uOZ6kRBJmzp9X2oB8MDHoKe4QEuiRx/5/DNxB8i2QoTWN/BfluTSfVpO5rf jSXlaUuFOnouBrWdmbaBdg+47m4IGEz129Zdf+y+ISexQ6P16ZY1oYxYlbQSaEwk0+TJ4B0C uvMHwPF3SDH2LeRx+mK2OvwnVulvj2+WdW/rIVgwhwbKmBLj40R+Uq4zD/4iRxJ5PF1ynjxR po3Izp/ZrYWrPgtBg0jUZ8DdlAiRHCFGPpccK8RvBWXmtzF4XQsV39aPBqcE3W6IcTnIMrDi 6mnqealpfiUq+4RGNfRFN9wtgViZLy/FRWi76k+vo/Jmp7/K9JblGX48D2JL9FX0w5PXkpE4 abmY1OASQUiwoJ4n1asxwEonSaWeYbI7X5IqdvevGyfYdSn4VEywdrYGtWjsWlZ/DPofPwsI bQXGY6o+wg9lDAk2L2nVTa05XuyOooUPwKLD0WrLOIxLmcbVv/tgJG03/uI4iDitSofTKnpz E+xdpfFIyw1Mb8PO4WJi0gpHmmLUbG8AMLS+8wSDFwIA4TXQFy9suXXzLuuzML+G5h9Mo5D6 q5HsIe59lhdwk7oEPZJ1NWLfLavTENQg5ObS2YT1KaFskFxxgtcU0aBytAxTjkgGRB8UunXl NJeCuTIAUxXw41P93V4Khigc5dEOG1kEDoq0dAlAE7AbL6Vzc/Go+UwivtUil3sXADOyM9PT JjLNnye+2V0ywQncJ1AG6sxICpPKzv8oYP6xwurEuKnF8DAWEHEwT+Fb277Idv1v8uMGvltp coe7olE0O+TRUtMEwtEp4g4m8ym1rJI/yfwXtHkS8QcVBA9LRqcWEna1VPlT1pk3BSq/1xQa F/4OLScBfV2JbF93sN0SLw== Message-ID: Date: Wed, 13 Nov 2019 11:26:48 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:55.0) Gecko/20100101 Thunderbird/55.0a1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 11/13/19 11:20, M. Manna wrote: > HI Mark, > > On Wed, 13 Nov 2019 at 15:38, Mark Thomas > wrote: > >> On 12/11/2019 19:11, M. Manna wrote: >>> HI Mark, >>> >>> following my previous reply, we have now confirmed that it's >>> indeed >> 8.5.45 >>> with APR 1.2.23 that's causing such high JVM CPU usage. We used >>> took out 2 out of 50 servers from the load balancer config, >>> reverted tomcat, and redeployed. With near to identical user >>> traffic, the two servers are responding normally >>> without/without traffic with 8.5.41. The JVM dump looks a lot >>> better with 8.5.41. >>> >>> We do think that the recent changes in APR and some other >>> tomcat jar may have caused compatibility issue on Windows >>> server 2016 (64-bit) platform. But unfortunately, we cannot >>> pinpoint exactly what change may have caused this (i.e. actual >>> OS vs Security Updates). With this in mind, we are also being >>> wary to move to 8.5.47 as we don't know if the same issue will >> occur >>> again. Since 8.5.41 has been packaged with previously accepted >> application >>> installer, we are more comfortable rolling back. >> >> Just to confirm, you see this high CPU usage with a clean install >> (no additional web applications deployed, no configuration >> changes) on Windows 2016 DataCenter (64-bit)? >> >> If this is the case, it should be fairly easy to reproduce. >> >> Mark >> >> We do not deploy multiple applications. In fact, Under tomcat > webapps/ROOT we only have one application (ours). Each tomcat > instance is hosted on a VM (total 50) and all of them are > identically configured (server.xml, web.xml, logging, CPU/RAM). We > have not made any other configuration change between 8.5.41 and > 8.5.45. And yes, I agree with you that it's fairly easy to > reproduce. I think the question is whether or not your application is required to be deployed. Can you reproduce this issue with just the stock applications bundled with Tomcat? - -chris -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3MLscACgkQHPApP6U8 pFip5A/+KOg4ZvATDli8zG9ZxniMoPdkOC9LQgVKscjeLZHL/A1gzVLP8UPZSiU7 1+p44WwJ5WGgwe8Ne0NZTFlh7/DZQAGIQZv++Ii9+NRkY5KVP3dYykdoyg1UdUMB Fdu2KNDcsCERYpPqrE/kVk+TQZNI60vY8iTBntc+Og5LsukULZTbX3UO9BzDaqeO WsjVuP6q7hUDBntd+5YqeFKDJ07zEIm5V6vmHAbCOWm3g2B8IXiYkMTXM+ZLld9h 6Th8f+na79taUrxT9TwI1WoR/ZJguJW1c8eRPbykv9/riDrtTQsv0BZy0ZeMhnjv kEwurNMaYjtSSCGOD0e8/chy1rU0/gng99pkmGe0Wiwoob6/6AU0HhE/2RvLKzDY mR4hu+aDaxtog4CD9DQrGenId+pwbJteqhXVCye6V0A3JtobbR+D56cxcUbth1pP skMdXrTWTvVlmsyLfKjPmMiALzOqg0bqvfYH5bEitW1Y8HvCQN5vcht2+EOpchmp zZ0f0LQpXEyr3DJ/GSbPTRKHghMAnrB4yz9jlMvzdWoPX2/JyT3+IQOoe8eRtlD8 e6uoQzXoguXFr9J5OLGR5TXdBLx5/obCUWUM2wS/w5TQ3MvV3C5haSKYWTVIcetp XuAzKmK6fKFBHn37pMLd4VELy9Ay+zQmtTrpDJzB9pPwX/gr6JQ= =VqkL -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org