From users-return-267574-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Tue May 21 14:55:46 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2851918061A for ; Tue, 21 May 2019 16:55:46 +0200 (CEST) Received: (qmail 3010 invoked by uid 500); 21 May 2019 14:55:43 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 2998 invoked by uid 99); 21 May 2019 14:55:43 -0000 Received: from Unknown (HELO mailrelay2-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 May 2019 14:55:43 +0000 Received: from Christophers-MacBook-Pro-2.local (pool-108-28-187-75.washdc.fios.verizon.net [108.28.187.75]) by mailrelay2-lw-us.apache.org (ASF Mail Server at mailrelay2-lw-us.apache.org) with ESMTPSA id 44ABB39B8 for ; Tue, 21 May 2019 14:55:39 +0000 (UTC) Subject: Re: Massive Startup Time after Server Reboot To: users@tomcat.apache.org References: <761e3fa1-6d82-9f36-9edd-9973af85285d@malcolms.com> <8ab39715-ebda-d743-5aaa-d57c04d1debf@malcolms.com> <5CE268CD.3030907@ice-sa.com> From: Christopher Schultz Openpgp: preference=signencrypt Autocrypt: addr=chris@christopherschultz.net; prefer-encrypt=mutual; keydata= mQINBE+pgz4BEADd7qAWgqXcNltlB3aow0UneRmNSVjHKgekgs0ZXxG9l50Athksr/3bL/yg bxFB00JcM9W+UxLhKHiMSyzfeBHn9l9wAlLFKs0S91KXTUnRwGFtvgstvGROoqPgTVREklnm yW/KpzOwqSrQ5xHcogaT+XWlXmRbtFypi52Z5HGWlFWWgwx0vKBWHmQayPtCif0v1RDxfdV9 zziodn0TnpfBQsEgf9TDAjkNT8f0ecwTnhSihTDm1W5HCK7Pm5DfUtree1Oh6Ncz2ljlUO0b 3Lai9pX48eZOj7WQXPefkcv2AoUvdELkQKw3klM5YNXbXPf1KAjky+q4DQ1ydD6LkK+9cI3S TeMesTlk/tytOsaN2NH2k87sEpcumbH0AcmPFEnIYUfm4KzWdKlYA6mbV3Pk3tHSuayyJovj h/7Y7BG9p2l7D60r49hzrTPG8VxNkSliNLcSjI3QjYpfhSlqmqXyVKzdzirK1HPr1xfJStig RpLP9nWarZjoXng9N0etGwtH/8roeDPYA8x9ba1KXy/1g/i+RLx2ms+rueCpnFZxU3GZNUSp RfpdUbwCN3Zm1w5Z6SI8X2aSnWWeYzU6HMsV+P4PROnFsgxDeOpyWhyEaaVLXQtOYwcHneHb n56vSG50TkAuHs5kk/3/YDPSsqjsUPOuhKgFMh3iqMTh5DMdSwARAQABtDJDaHJpc3RvcGhl ciBTY2h1bHR6IDxjaHJpc0BjaHJpc3RvcGhlcnNjaHVsdHoubmV0PokCOgQTAQgAJAIbLwUL CQgHAwUVCgkICwUWAgMBAAIeAQIXgAUCT6mETgIZAQAKCRDzrVyUpn9wflkxD/9IsahRqHTV /hH5nuPqVO692cQqHvPtMPO6lDb4909VN5T1i+1hFr80P0KVDL6EI78lDBJ2TThWI0o5vFdm sRlei59wsgTvkKTph5QwwOWl7OyzUDX3WbKhkNQdGf4I+/g/1s2bHaRoG30ELdL7cwUPCPrW 0KQwBy7Rtr0WbdujKOw9b/UcgyXEOE1wNcorq/E1o5/6BRYIcFQOO4sjHjGcChOpSg5ms4zb s+Xv3gOtLrbmOPRTXdvBxwJA6kkfQFHvI42kXYghTdqhBVPnHYPqUeavRsb+Yz3ghkZhj35i GfaGyXNwFBikCYjzIaj44NOkT1pU50MgIbjSJ+xoHnC20T942kekqp6wzqUM19Pa9ohsEdA1 Sf6/A7RmpZRrxSIY02ZVnGccnVjglnylVcnxrNAZC3ebxCeZPQ09FBR0Uqlsrdt7A3hlEP2F aoMTSa+hYqfWBGB7uZhcJZIsZspxm8J0txeOzYNSFDl7mF134ShRsq6dpSugCdcdeSWKliBz q0U8sIabOFLMxM0hbwkn2RG4OaurJLWXQf+7IhA/J8TizjkbdxLmR2PiTiVtrx484mpWpbF8 po/em0q/reFnL+JtOM6qlJE/Q4B6PfkchhU5vKPfmGw98t9guyw5G8YSR1rR+SOowHg4T/i2 Rezz1idKmoFpPdNFRPlOAC+d67kCDQRPqYM+ARAAzEItVpzvcgZB+faUWi54lJoA8GnVxXEe OQY+7wk/P5i9GtL0UVXC53j2F87BDVXGalKgVjEVdNY3Cyx+dJ2os65gjxd6ZK18zc6N7YZB Z00XNU9nTz5XImZzHn4VmeXYMQrKO/981nCNPlV6CVdgGg9wl1Ij5Sh8SSTb8kWSo1ngx+XX 4yJNUbfSh32yMPVGI7ZcoZLm9gdgTOOnuEkeeGs/lPvYN+1Cv/YtvkPybSOSWSdHxIVU4Iko 6V7IkM1amjdwKfoeg+CLhZsbY7VLAzVtGvaF5z4rtJtCfTfhbYD0wS8afEBcvsew1HdtYDT5 AJqojeZBGDuY7JCgALc3HCy34Zzk+mi1qwvrm5i/CBMuIvjxB2MkzhHQNUD20fzdRcoIgw4J IzbqZLlOpVFehDXzKT/h5vh+Uv7s6Rz5gP5i0Rkcghw00mRBvuN8mpQnLt4hYL22cNh/tk0L Fxda7ZaPehu7ug4E5FEB0Ifm1KV18P7Kpfu8tiSLz7rl++x73o4uv4bk1ZnjO/jFsx0KLGwq VxR276ZIwsV4WpLYvJ5fR0kqqd/TOKXGSEA0eGxWTeb/fNtkYemRVoasB1+rqjh/Rz0p20o8 elkqDhpzzhrMNzEMYkLySu7npWCeWW4Nd6097+OG9BCLO+ndGmAcupdu6WMEj2UlWsQxuCYC PgsAEQEAAYkEPgQYAQgACQUCT6mDPgIbLgIpCRDzrVyUpn9wfsFdIAQZAQgABgUCT6mDPgAK CRAc8Ck/pTykWO6WD/0XlAG4D4GwzzuOfh7DG6cm/I0vmASEJkY5ghStW4GUbYosgS/btyj/ YPWzVh4HWMvuA6YYKCuz/CM3h34dR25XmHqUdOyJOCnMJ3psdv5YsytgnEdvINZALlDdBX3G sfytgS0KnVjAc92LfJOxHAsZf4zE3SU28FMX7jCgeqO3YrvkHsZ8dzzgw3QYT0J3NcYfkflb DPBXBDGrvdUuea/w6F17pctdRdt7jE3JiLFq2F9ehXOSsIwecUlqVYiCRuxblD4cJ6gKMn0y 8zllW4GyIbf/+mNLkpKoMPYnptDvcEojluHtwbkSfF5AwgJbm6pfs9a2vpGBVko+dBXGh4/T 3qNYxeGEAsI0psEJu3EZN9dYv/ZOb69DUJ6SwEKp/L7lU7C8HoLx/MpKtuJO9OS5uuAhdBSi GqfaN9zP2NxPXSwnexVK2exy/h5sUevDsnBEHmyxe5GRSrIilyijLtlYhq2W7G95poxIFZuL Db98R+7VR9Yl9uOZ6kRBJmzp9X2oB8MDHoKe4QEuiRx/5/DNxB8i2QoTWN/BfluTSfVpO5rf jSXlaUuFOnouBrWdmbaBdg+47m4IGEz129Zdf+y+ISexQ6P16ZY1oYxYlbQSaEwk0+TJ4B0C uvMHwPF3SDH2LeRx+mK2OvwnVulvj2+WdW/rIVgwhwbKmBLj40R+Uq4zD/4iRxJ5PF1ynjxR po3Izp/ZrYWrPgtBg0jUZ8DdlAiRHCFGPpccK8RvBWXmtzF4XQsV39aPBqcE3W6IcTnIMrDi 6mnqealpfiUq+4RGNfRFN9wtgViZLy/FRWi76k+vo/Jmp7/K9JblGX48D2JL9FX0w5PXkpE4 abmY1OASQUiwoJ4n1asxwEonSaWeYbI7X5IqdvevGyfYdSn4VEywdrYGtWjsWlZ/DPofPwsI bQXGY6o+wg9lDAk2L2nVTa05XuyOooUPwKLD0WrLOIxLmcbVv/tgJG03/uI4iDitSofTKnpz E+xdpfFIyw1Mb8PO4WJi0gpHmmLUbG8AMLS+8wSDFwIA4TXQFy9suXXzLuuzML+G5h9Mo5D6 q5HsIe59lhdwk7oEPZJ1NWLfLavTENQg5ObS2YT1KaFskFxxgtcU0aBytAxTjkgGRB8UunXl NJeCuTIAUxXw41P93V4Khigc5dEOG1kEDoq0dAlAE7AbL6Vzc/Go+UwivtUil3sXADOyM9PT JjLNnye+2V0ywQncJ1AG6sxICpPKzv8oYP6xwurEuKnF8DAWEHEwT+Fb277Idv1v8uMGvltp coe7olE0O+TRUtMEwtEp4g4m8ym1rJI/yfwXtHkS8QcVBA9LRqcWEna1VPlT1pk3BSq/1xQa F/4OLScBfV2JbF93sN0SLw== Message-ID: Date: Tue, 21 May 2019 10:55:38 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Jerry, On 5/20/19 14:05, Jerry Malcolm wrote: > Andre, > > Your theory sounds like a real possibility. It would explain why > it only blows up the first time after a reboot. I'll test the > theory tonight during off-peak use hours for the web sites I host. If André's theory is correct, you will probably find that 2-3 months ago the version of your virus scanner was upgraded, and all of the whitelisted applications (like Tomcat) were lost. If that's the case, you can probably re-whitelist Tomcat and get your performance back. At least until the next upgrade. - -chris > On 5/20/2019 3:43 AM, André Warnier (tomcat) wrote: >> On 20.05.2019 00:09, Jerry Malcolm wrote: >>> Just an FYI.... this server has been in production a little >>> over a year. This TC load problem only started 2-3 months ago. >>> So 'something' changed to cause this problem to manifest >>> itself. It's been too long to try to correlate a specific >>> server change to the start of this problem. But I just wanted >>> to point out that this problem has NOT been around since day 1 >>> on this server. >> >> Just in case, since this is a Windows server .. 2-3 months ago, >> an update of a virus scanner ? (of the intrusive kind which >> pre-checks each file that wants to be opened) Checking may be >> easy or not, depending on your access : disable the virus scanner >> just the time to start tomcat and check. >> >>> >>> Jerry >>> >>> On 5/19/2019 5:03 PM, Jerry Malcolm wrote: >>>> Rainer, >>>> >>>> No change with the urandom parm. I am attaching a portion of >>>> the Catalina log. The first half shows between 8 and 15 >>>> seconds to deploy each app for a single virtual host (there >>>> are no war files, the app is already exploded in the appbase >>>> dir). I have quite a few virtual hosts with several apps >>>> each. That initial server start took 21+ minutes. I then >>>> restarted the TC service and got the ~.5 sec start per webapp >>>> shown in the 2nd half of the log below. >>>> >>>> Can you refresh me on how to capture the stack dumps you >>>> suggested? It's been a while.... >>>> >>>> Thx. >>>> >>>> Jerry >>>> >>>> First start of Tomcat after server reboot >>>> ------------------------------------------------------------------- - -- >>>> >>>> [C:\domains\.com\webapps\JSPWiki.war] has finished in >>>> [8,579] ms tory [C:\domains\.com\webapps\cis] >>>> irectory [C:\domains\.com\webapps\cis] has finished >>>> in [11,486] ms tory >>>> [C:\domains\.com\webapps\gallery] irectory >>>> [C:\domains\.com\webapps\gallery] has finished in >>>> [9,204] ms tory [C:\domains\.com\webapps\gl] >>>> irectory [C:\domains\.com\webapps\gl] has finished >>>> in [8,469] ms tory >>>> [C:\domains\.com\webapps\idmanager] irectory >>>> [C:\domains\.com\webapps\idmanager] has finished in >>>> [8,689] ms tory >>>> [C:\domains\.com\webapps\itemtrack] irectory >>>> [C:\domains\.com\webapps\itemtrack] has finished in >>>> [6,907] ms tory >>>> [C:\domains\.com\webapps\malcolment] irectory >>>> [C:\domains\.com\webapps\malcolment] has finished >>>> in [8,469] ms tory >>>> [C:\domains\.com\webapps\notebook] irectory >>>> [C:\domains\.com\webapps\notebook] has finished in >>>> [10,189] ms tory [C:\domains\.com\webapps\order] >>>> irectory [C:\domains\.com\webapps\order] has >>>> finished in [8,501] ms tory >>>> [C:\domains\.com\webapps\payment] irectory >>>> [C:\domains\.com\webapps\payment] has finished in >>>> [14,209] ms tory >>>> [C:\domains\.com\webapps\projectmanager] irectory >>>> [C:\domains\.com\webapps\projectmanager] has >>>> finished in [9,018] ms [C:\Tomcat >>>> 9.0\conf\Catalina\myridetx.net\manager.xml] ptor [C:\Tomcat >>>> 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in >>>> [62] ms >>>> >>>> Restart of Tomcat >>>> ------------------------------------------------------------------- - -- >>>> >>>> tory [C:\domains\.com\webapps\cis] >>>> irectory [C:\domains\.com\webapps\cis] has finished >>>> in [594] ms tory [C:\domains\.com\webapps\gallery] >>>> irectory [C:\domains\.com\webapps\gallery] has >>>> finished in [547] ms tory >>>> [C:\domains\.com\webapps\gl] irectory >>>> [C:\domains\.com\webapps\gl] has finished in [562] >>>> ms tory [C:\domains\.com\webapps\idmanager] >>>> irectory [C:\domains\.com\webapps\idmanager] has >>>> finished in [578] ms tory >>>> [C:\domains\.com\webapps\itemtrack] irectory >>>> [C:\domains\.com\webapps\itemtrack] has finished in >>>> [547] ms tory [C:\domains\.com\webapps\malcolment] >>>> irectory [C:\domains\.com\webapps\malcolment] has >>>> finished in [579] ms tory >>>> [C:\domains\.com\webapps\notebook] irectory >>>> [C:\domains\.com\webapps\notebook] has finished in >>>> [547] ms tory [C:\domains\.com\webapps\order] >>>> irectory [C:\domains\.com\webapps\order] has >>>> finished in [531] ms tory >>>> [C:\domains\.com\webapps\payment] irectory >>>> [C:\domains\.com\webapps\payment] has finished in >>>> [562] ms tory >>>> [C:\domains\.com\webapps\projectmanager] irectory >>>> [C:\domains\.com\webapps\projectmanager] has >>>> finished in [531] ms >>>> >>>> On 5/18/2019 3:18 PM, Rainer Jung wrote: >>>>> Most likely it hangs waiting for enough entropy for random >>>>> number generator seeding. >>>>> >>>>> Try whether the problem goes away if you add >>>>> >>>>> -Djava.security.egd=file:/dev/urandom >>>>> >>>>> to you process flags. If you are using older Java than Java >>>>> 8 (not possible for Tomcat 9 but just in case you also have >>>>> older software stacks running), then it would be >>>>> >>>>> -Djava.security.egd=file:/dev/./urandom >>>>> >>>>> The cryptic /./ is not a typo, you would need it for Java >>>>> before Java 8. >>>>> >>>>> If this doesn't help, then you should try capturing a few >>>>> stack dumps (thread dumps) during the long startup time. >>>>> People here can help interpret them. >>>>> >>>>> Regards, >>>>> >>>>> Rainer >>>>> >>>>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm: >>>>>> This is a weird one. It started a few months ago. I >>>>>> have TC 9 running on Windows Server 16. After I reboot >>>>>> the entire server, Tomcat takes forever on startup. It >>>>>> normally starts in about 30 seconds. But after a server >>>>>> reboot it takes up to 15 minutes... chugging along at a >>>>>> snail's pace starting up all of the apps on all of the >>>>>> virtual hosts. It always finally gets there with >>>>>> everything successfully running. Other servers on the >>>>>> same box (Apache, JAMES, ISC BIND, MySQL) don't have any >>>>>> problem starting up. CPU, Disk, Memory, etc. usages are >>>>>> barely showing on the performance graphs. There's nothing >>>>>> in the Catalina log or system.err other than showing a >>>>>> couple of minute gap in the time stamp between each app >>>>>> as it starts up. If I need to reboot again later, it >>>>>> boots again in about 30 sec as expected. >>>>>> >>>>>> First question... any ideas off the top of your head that >>>>>> might be causing this? >>>>>> >>>>>> Second question... is there any other logging I can turn >>>>>> on that might explain what TC is blocking on? >>>>>> >>>>>> Thanks. >>>>>> >>>>>> Jerry >>>>> >>>>> ------------------------------------------------------------------ - --- >>>>> >>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>> For additional commands, e-mail: >>>>> users-help@tomcat.apache.org >>>>> >>>> >>>> ------------------------------------------------------------------- - -- >>>> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>> For additional commands, e-mail: >>>> users-help@tomcat.apache.org >>>> >>> >>> -------------------------------------------------------------------- - - >>> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>> For additional commands, e-mail: users-help@tomcat.apache.org >>> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >> For additional commands, e-mail: users-help@tomcat.apache.org >> > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org > For additional commands, e-mail: users-help@tomcat.apache.org > -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlzkEWoACgkQHPApP6U8 pFinLRAAttjrmJnG6lwB3pd7kT5IpBk5H/MwJdXvEX8/S/k6MLLgWqWmqQhgRQAZ UeVqJFJFDCrgXymxXNI2f7k3saVA+2Lxafjuft9WQGKs+mTSQR0s82VG6Iko5JzL dZxA5XD+SSU6PafTMQ1hmlVMMDeug+ONqMcUY2XFoZSV6TUk6KqjMiod74We7/b8 7PSqI4VWSS6fA4/ZoPJNRvGLXtYDu1obVX3YysMb4tSky/5OenJX6rz0mwMenY0z F2p2G2ObNlLKUq7DuAhBcLHB4lbkEv+20Zv6hqMGPlFB8rjiOpUpL86/gfdJY29z KGgWLAw1M4rw2n0GjVihcfYfjVBA931wWwkaXpa6cXCrcyjOcgCH+OlhIqjtoHkW w6mM9m99LqrWRwqqfiHDDxqywdqukfGANwi1QvaPgVkd0+FwGkinh5G3P9au3cLs JKdpGgEog1hJnjkkXWwROlozKiKjMD82V474E4mwjw7nqPI/YXXL5E8VTlhm11JB OWDZ8yNgwA//Aw7JbGdAUPqRSsSqrKIyGYr4voDd6nhs+83R/7z27IyIxgw/kzFU 6xRFOorztK5dOR5TqPZYB67qUL39++fpdayTYDs7f5Y8guajOsWgVTrvWo3bxCkg CxUE7V91tWJYJKlxgbFZyZp4vvSzabf4dq5fxKhPTkYH9pmHhCs= =z1un -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org