From users-return-265618-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Wed Sep 26 21:46:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6F338180629 for ; Wed, 26 Sep 2018 21:46:03 +0200 (CEST) Received: (qmail 73503 invoked by uid 500); 26 Sep 2018 19:46:01 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 73492 invoked by uid 99); 26 Sep 2018 19:46:01 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Sep 2018 19:46:01 +0000 Received: from Christophers-MacBook-Pro-2.local (marriott-chateau-champlain-montreal.sites.intello.com [66.171.169.34]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id B6865E09 for ; Wed, 26 Sep 2018 19:46:00 +0000 (UTC) Subject: Re: Application hanging on Tomcat 7.0.54 To: users@tomcat.apache.org References: From: Christopher Schultz Openpgp: preference=signencrypt Autocrypt: addr=chris@christopherschultz.net; prefer-encrypt=mutual; keydata= xsFNBE+pgz4BEADd7qAWgqXcNltlB3aow0UneRmNSVjHKgekgs0ZXxG9l50Athksr/3bL/yg bxFB00JcM9W+UxLhKHiMSyzfeBHn9l9wAlLFKs0S91KXTUnRwGFtvgstvGROoqPgTVREklnm yW/KpzOwqSrQ5xHcogaT+XWlXmRbtFypi52Z5HGWlFWWgwx0vKBWHmQayPtCif0v1RDxfdV9 zziodn0TnpfBQsEgf9TDAjkNT8f0ecwTnhSihTDm1W5HCK7Pm5DfUtree1Oh6Ncz2ljlUO0b 3Lai9pX48eZOj7WQXPefkcv2AoUvdELkQKw3klM5YNXbXPf1KAjky+q4DQ1ydD6LkK+9cI3S TeMesTlk/tytOsaN2NH2k87sEpcumbH0AcmPFEnIYUfm4KzWdKlYA6mbV3Pk3tHSuayyJovj h/7Y7BG9p2l7D60r49hzrTPG8VxNkSliNLcSjI3QjYpfhSlqmqXyVKzdzirK1HPr1xfJStig RpLP9nWarZjoXng9N0etGwtH/8roeDPYA8x9ba1KXy/1g/i+RLx2ms+rueCpnFZxU3GZNUSp RfpdUbwCN3Zm1w5Z6SI8X2aSnWWeYzU6HMsV+P4PROnFsgxDeOpyWhyEaaVLXQtOYwcHneHb n56vSG50TkAuHs5kk/3/YDPSsqjsUPOuhKgFMh3iqMTh5DMdSwARAQABzTJDaHJpc3RvcGhl ciBTY2h1bHR6IDxjaHJpc0BjaHJpc3RvcGhlcnNjaHVsdHoubmV0PsLBegQTAQgAJAIbLwUL CQgHAwUVCgkICwUWAgMBAAIeAQIXgAUCT6mETgIZAQAKCRDzrVyUpn9wflkxD/9IsahRqHTV /hH5nuPqVO692cQqHvPtMPO6lDb4909VN5T1i+1hFr80P0KVDL6EI78lDBJ2TThWI0o5vFdm sRlei59wsgTvkKTph5QwwOWl7OyzUDX3WbKhkNQdGf4I+/g/1s2bHaRoG30ELdL7cwUPCPrW 0KQwBy7Rtr0WbdujKOw9b/UcgyXEOE1wNcorq/E1o5/6BRYIcFQOO4sjHjGcChOpSg5ms4zb s+Xv3gOtLrbmOPRTXdvBxwJA6kkfQFHvI42kXYghTdqhBVPnHYPqUeavRsb+Yz3ghkZhj35i GfaGyXNwFBikCYjzIaj44NOkT1pU50MgIbjSJ+xoHnC20T942kekqp6wzqUM19Pa9ohsEdA1 Sf6/A7RmpZRrxSIY02ZVnGccnVjglnylVcnxrNAZC3ebxCeZPQ09FBR0Uqlsrdt7A3hlEP2F aoMTSa+hYqfWBGB7uZhcJZIsZspxm8J0txeOzYNSFDl7mF134ShRsq6dpSugCdcdeSWKliBz q0U8sIabOFLMxM0hbwkn2RG4OaurJLWXQf+7IhA/J8TizjkbdxLmR2PiTiVtrx484mpWpbF8 po/em0q/reFnL+JtOM6qlJE/Q4B6PfkchhU5vKPfmGw98t9guyw5G8YSR1rR+SOowHg4T/i2 Rezz1idKmoFpPdNFRPlOAC+d687BTQRPqYM+ARAAzEItVpzvcgZB+faUWi54lJoA8GnVxXEe OQY+7wk/P5i9GtL0UVXC53j2F87BDVXGalKgVjEVdNY3Cyx+dJ2os65gjxd6ZK18zc6N7YZB Z00XNU9nTz5XImZzHn4VmeXYMQrKO/981nCNPlV6CVdgGg9wl1Ij5Sh8SSTb8kWSo1ngx+XX 4yJNUbfSh32yMPVGI7ZcoZLm9gdgTOOnuEkeeGs/lPvYN+1Cv/YtvkPybSOSWSdHxIVU4Iko 6V7IkM1amjdwKfoeg+CLhZsbY7VLAzVtGvaF5z4rtJtCfTfhbYD0wS8afEBcvsew1HdtYDT5 AJqojeZBGDuY7JCgALc3HCy34Zzk+mi1qwvrm5i/CBMuIvjxB2MkzhHQNUD20fzdRcoIgw4J IzbqZLlOpVFehDXzKT/h5vh+Uv7s6Rz5gP5i0Rkcghw00mRBvuN8mpQnLt4hYL22cNh/tk0L Fxda7ZaPehu7ug4E5FEB0Ifm1KV18P7Kpfu8tiSLz7rl++x73o4uv4bk1ZnjO/jFsx0KLGwq VxR276ZIwsV4WpLYvJ5fR0kqqd/TOKXGSEA0eGxWTeb/fNtkYemRVoasB1+rqjh/Rz0p20o8 elkqDhpzzhrMNzEMYkLySu7npWCeWW4Nd6097+OG9BCLO+ndGmAcupdu6WMEj2UlWsQxuCYC PgsAEQEAAcLDfgQYAQgACQUCT6mDPgIbLgIpCRDzrVyUpn9wfsFdIAQZAQgABgUCT6mDPgAK CRAc8Ck/pTykWO6WD/0XlAG4D4GwzzuOfh7DG6cm/I0vmASEJkY5ghStW4GUbYosgS/btyj/ YPWzVh4HWMvuA6YYKCuz/CM3h34dR25XmHqUdOyJOCnMJ3psdv5YsytgnEdvINZALlDdBX3G sfytgS0KnVjAc92LfJOxHAsZf4zE3SU28FMX7jCgeqO3YrvkHsZ8dzzgw3QYT0J3NcYfkflb DPBXBDGrvdUuea/w6F17pctdRdt7jE3JiLFq2F9ehXOSsIwecUlqVYiCRuxblD4cJ6gKMn0y 8zllW4GyIbf/+mNLkpKoMPYnptDvcEojluHtwbkSfF5AwgJbm6pfs9a2vpGBVko+dBXGh4/T 3qNYxeGEAsI0psEJu3EZN9dYv/ZOb69DUJ6SwEKp/L7lU7C8HoLx/MpKtuJO9OS5uuAhdBSi GqfaN9zP2NxPXSwnexVK2exy/h5sUevDsnBEHmyxe5GRSrIilyijLtlYhq2W7G95poxIFZuL Db98R+7VR9Yl9uOZ6kRBJmzp9X2oB8MDHoKe4QEuiRx/5/DNxB8i2QoTWN/BfluTSfVpO5rf jSXlaUuFOnouBrWdmbaBdg+47m4IGEz129Zdf+y+ISexQ6P16ZY1oYxYlbQSaEwk0+TJ4B0C uvMHwPF3SDH2LeRx+mK2OvwnVulvj2+WdW/rIVgwhwbKmBLj40R+Uq4zD/4iRxJ5PF1ynjxR po3Izp/ZrYWrPgtBg0jUZ8DdlAiRHCFGPpccK8RvBWXmtzF4XQsV39aPBqcE3W6IcTnIMrDi 6mnqealpfiUq+4RGNfRFN9wtgViZLy/FRWi76k+vo/Jmp7/K9JblGX48D2JL9FX0w5PXkpE4 abmY1OASQUiwoJ4n1asxwEonSaWeYbI7X5IqdvevGyfYdSn4VEywdrYGtWjsWlZ/DPofPwsI bQXGY6o+wg9lDAk2L2nVTa05XuyOooUPwKLD0WrLOIxLmcbVv/tgJG03/uI4iDitSofTKnpz E+xdpfFIyw1Mb8PO4WJi0gpHmmLUbG8AMLS+8wSDFwIA4TXQFy9suXXzLuuzML+G5h9Mo5D6 q5HsIe59lhdwk7oEPZJ1NWLfLavTENQg5ObS2YT1KaFskFxxgtcU0aBytAxTjkgGRB8UunXl NJeCuTIAUxXw41P93V4Khigc5dEOG1kEDoq0dAlAE7AbL6Vzc/Go+UwivtUil3sXADOyM9PT JjLNnye+2V0ywQncJ1AG6sxICpPKzv8oYP6xwurEuKnF8DAWEHEwT+Fb277Idv1v8uMGvltp coe7olE0O+TRUtMEwtEp4g4m8ym1rJI/yfwXtHkS8QcVBA9LRqcWEna1VPlT1pk3BSq/1xQa F/4OLScBfV2JbF93sN0SLw== Message-ID: <333c0ced-fe6d-a7d3-6716-d885b6e8ea10@christopherschultz.net> Date: Wed, 26 Sep 2018 15:46:00 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Louis, On 9/26/18 14:42, Louis Zipes wrote: > Hi all, Tomcat 7.0.54 running on Windows 2012 > > We are running a third party application on Tomcat and today we > have intermittently run in issues where the application stops > working. The big changes in our system is that we have added more > end users and we are at year end so of course everyone is hitting > the system hard. Even if we force a log out of all users and stop > all background jobs then the application doesn't recover. > > We see no active sessions on the database (our application is > connecting to an Oracle database) and I see no clear error messages > in either our third party application logs or the Tomcat logs (ex. > OutofMemory). When we go to the Windows Task Manager we did not > see the machine's Memory max'd out but admittedly I didn't look at > the Java session to see if was reaching its Heap Max. The only > thing that we noticed was that TCP connections went down right > after the restart. I did open up Jconsole under Java and I did > force a garbage collection but that didn't seem to help. > > We do have an Oracle Grid Control and we did get an alert in > regards to Metric: [HTTP Transaction] Perceived Time per Page going > past thresholds but not sure if that was just an old alert with and > old range that was set up a long time ago or is a really valid > clue. Since this is PRD we had to get it back up and running so > all I did was increase the Tomcat Xmx Heap size and restarted. I'm > not really confident that is the solution since as mentioned you > tend to see a clear out of memory error if it was too small. > > So a few questions: > > > 1) Does this sound like a known issue with this earlier version > of Tomcat? No. > 2) Should I turn up any logging on Tomcat and if so which > ones? Not yet. > 3) We didn't do a JSTACK dump while it was happening. Would > that have been useful? Absolutely. > 4) Do we need to play around with MaxThreads and/or > MaxConnections. We do have maxThreads in our server.mxl but in DEV > when we turned it down to a value = 5 hoping to overwhelm it > nothing bad happened. Don't change anything, yet. > Once again, we are limited to what we could do and collect since it > was PRD and we needed to restart it. We restarted the Tomcat > service and everything is processing fine for right now. I will > note that that we did have that bad Windows patch that prevented it > from stopping and starting cleanly > (https://stackoverflow.com/questions/51498291/tomcat-lockup-on-shutdow n) > but we have taken the break fix patch and the daily restarts seem > to be fine since then. > > Any comments/suggestions are appreciated! Your most likely problem is database connection pool mismanagement: connections aren't properly released and the pool empties. All threads are left waiting on available database connections which will never be replenished. - -chris -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlur4fgACgkQHPApP6U8 pFhq/RAAwZixHqbHxYdX3VCrTfvz0tnOmu7W4sbeFqhExV+M6NVL2LK1RO26eTq5 OJB2o/RheCeajWHxqiJQY4ERMTOyyqZYCsRG9L901heW2UAW122zeX7hhXDB1IMo qIBVYQalg1j5e2Lw9MqT3ISj6U/GNA6VlirTAHGtuEBBpKXXyb6KtmOgpjHjjXS7 mTYni2iTHO/NaGrS519alFPMBnF4Wq5NzRcLewNMqj9Nbx2uu3Suu95DhJ+WIIep DOmyy4UGHxGx2QqNUnVWMHApGGjFD4pTWIBwzcbbsL56kZDxRGsF0SsB0VR/jSMY HdI71RgjpQFSyar1rcCTPCRP97KnUC+oaKhn+i2jBMSRxs94GEOqk4LuKNo39HKP tXEMkYl0o/CR9QaFDjncy9M2M3/o50ooBvYTOu0SjmyZO+ab9tJpaXbnlf2ChxJI AWbhBGmwJ6kF5FvVbmzujV7EEF2YCRMBpWo2zjNd6zWvX9OWEOXrOaHuX7iBPCga YqEMQWyS7XWiBG7AI8+ka5x4s/oxWsbn/6pCdDXhfxl5p5jv7ajm7LbLXGut1N6a uV5hmLLgZywF68AYe6X3GWv6mygXMBYABZxEA6klWE7HpIzvLxmxu0+vFlYR8qsb FMAacJ/FYJ9nGRuMqG+V2Edr5U//JvrWqy4raPIvwXGtX+FsqEc= =Zavb -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org