From users-return-267581-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Tue May 21 18:52:53 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7E45518061A for ; Tue, 21 May 2019 20:52:53 +0200 (CEST) Received: (qmail 92035 invoked by uid 500); 21 May 2019 18:52:48 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 92020 invoked by uid 99); 21 May 2019 18:52:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 May 2019 18:52:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 82D4FC2CF2 for ; Tue, 21 May 2019 18:52:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.001 X-Spam-Level: * X-Spam-Status: No, score=1.001 tagged_above=-999 required=6.31 tests=[DKIM_INVALID=0.1, DKIM_SIGNED=0.1, KAM_ASCII_DIVIDERS=0.8, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=fail (1024-bit key) reason="fail (body has been altered)" header.d=jwmhosting.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 2KVYzoiNj4_1 for ; Tue, 21 May 2019 18:52:42 +0000 (UTC) Received: from mx3.jwmhosting.com (jwmhosting.com [64.34.196.231]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id AB749612BE for ; Tue, 21 May 2019 18:52:41 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit DKIM-Signature: a=rsa-sha256; b=NGdZ382PYT8/HFxrwFmm9XgGbztiBW9ADcro14cM7AYm+IThovUrXkXPlrqqGn69fNvhafluOzYxWKKFukRrjlwNZrp0+nyYIgg+sc9lPz9ExEh3FKmSkhjW/+tmo3l24B/BgbI3/qXfGXuGElfwn6GKUB18cJNQoNQFHiaX/J4=; s=primary; c=relaxed/relaxed; d=jwmhosting.com; v=1; bh=+44c1/bF0kjHzF3WXFLVyCjiBj7JxGLin4SbXbvUO+E=; h=Message-ID:Date:Subject:From:To:MIME-Version:Content-Type; X-Processor-Root: true X-Processor-Authorized-Outbound: true X_JWMH_OUTBOUND: true X_JWMH_FLAG: $label3 X_JWMH_TGTFOLDER: Apache Forums.Tomcat Users X-Processor-Transport: true X-UserIsAuth: true X-MIME-Autoconverted: from 8bit to quoted-printable by Apache JAMES Received: from cpe-66-68-42-68.austin.res.rr.com (EHLO [192.168.0.6]) ([66.68.42.68]) by jwmhosting.com (JAMES SMTP Server ) with ESMTPA ID 1013752645 for ; Tue, 21 May 2019 13:52:33 -0500 (CDT) Subject: Re: Massive Startup Time after Server Reboot To: users@tomcat.apache.org References: <761e3fa1-6d82-9f36-9edd-9973af85285d@malcolms.com> <8ab39715-ebda-d743-5aaa-d57c04d1debf@malcolms.com> <5CE268CD.3030907@ice-sa.com> <8359f27a-7524-d5aa-15ea-035815562a83@malcolms.com> From: Jerry Malcolm Message-ID: Date: Tue, 21 May 2019 13:52:31 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 In-Reply-To: Content-Language: en-US It's currently automatic.  I'll try delayed-start tonight and see if there are changes. Thx On 5/21/2019 1:21 PM, Suvendu Sekhar Mondal wrote: > Jerry, > > > On Tue, May 21, 2019, 10:55 PM Jerry Malcolm >> Update.... I totally disabled Windows Real Time Virus protection. >> Apparently it was 'one' of the problems. However... >> >> Original startup after reboot (with WIndows Real Time Virus scan) -- ~21 >> minutes >> >> With Windows Real Time Virus scan totally disabled immediately after >> reboot -- ~3 minutes >> > Wow! > >> Second TC startup -- ~35 seconds. >> >> 3 minutes beats 21 minutes, and I guess I can live with it. But there's >> still something else slowing it down on the first startup after system >> reboot. I'll see about getting some stack dumps. >> > May I know what is the startup type of Tomcat service? If it is > 'automatic', can you please try with 'automatic delayed start'? > >> Thanks for the help. >> >> Jerry >> >> >> On 5/21/2019 9:55 AM, Christopher Schultz wrote: >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA256 >>> >>> Jerry, >>> >>> On 5/20/19 14:05, Jerry Malcolm wrote: >>>> Andre, >>>> >>>> Your theory sounds like a real possibility. It would explain why >>>> it only blows up the first time after a reboot. I'll test the >>>> theory tonight during off-peak use hours for the web sites I host. >>> If André's theory is correct, you will probably find that 2-3 months >>> ago the version of your virus scanner was upgraded, and all of the >>> whitelisted applications (like Tomcat) were lost. >>> >>> If that's the case, you can probably re-whitelist Tomcat and get your >>> performance back. At least until the next upgrade. >>> >>> - -chris >>> >>>> On 5/20/2019 3:43 AM, André Warnier (tomcat) wrote: >>>>> On 20.05.2019 00:09, Jerry Malcolm wrote: >>>>>> Just an FYI.... this server has been in production a little >>>>>> over a year. This TC load problem only started 2-3 months ago. >>>>>> So 'something' changed to cause this problem to manifest >>>>>> itself. It's been too long to try to correlate a specific >>>>>> server change to the start of this problem. But I just wanted >>>>>> to point out that this problem has NOT been around since day 1 >>>>>> on this server. >>>>> Just in case, since this is a Windows server .. 2-3 months ago, >>>>> an update of a virus scanner ? (of the intrusive kind which >>>>> pre-checks each file that wants to be opened) Checking may be >>>>> easy or not, depending on your access : disable the virus scanner >>>>> just the time to start tomcat and check. >>>>> >>>>>> Jerry >>>>>> >>>>>> On 5/19/2019 5:03 PM, Jerry Malcolm wrote: >>>>>>> Rainer, >>>>>>> >>>>>>> No change with the urandom parm. I am attaching a portion of >>>>>>> the Catalina log. The first half shows between 8 and 15 >>>>>>> seconds to deploy each app for a single virtual host (there >>>>>>> are no war files, the app is already exploded in the appbase >>>>>>> dir). I have quite a few virtual hosts with several apps >>>>>>> each. That initial server start took 21+ minutes. I then >>>>>>> restarted the TC service and got the ~.5 sec start per webapp >>>>>>> shown in the 2nd half of the log below. >>>>>>> >>>>>>> Can you refresh me on how to capture the stack dumps you >>>>>>> suggested? It's been a while.... >>>>>>> >>>>>>> Thx. >>>>>>> >>>>>>> Jerry >>>>>>> >>>>>>> First start of Tomcat after server reboot >>>>>>> ------------------------------------------------------------------- >>> - -- >>> [C:\domains\.com\webapps\JSPWiki.war] has finished in >>>>>>> [8,579] ms tory [C:\domains\.com\webapps\cis] >>>>>>> irectory [C:\domains\.com\webapps\cis] has finished >>>>>>> in [11,486] ms tory >>>>>>> [C:\domains\.com\webapps\gallery] irectory >>>>>>> [C:\domains\.com\webapps\gallery] has finished in >>>>>>> [9,204] ms tory [C:\domains\.com\webapps\gl] >>>>>>> irectory [C:\domains\.com\webapps\gl] has finished >>>>>>> in [8,469] ms tory >>>>>>> [C:\domains\.com\webapps\idmanager] irectory >>>>>>> [C:\domains\.com\webapps\idmanager] has finished in >>>>>>> [8,689] ms tory >>>>>>> [C:\domains\.com\webapps\itemtrack] irectory >>>>>>> [C:\domains\.com\webapps\itemtrack] has finished in >>>>>>> [6,907] ms tory >>>>>>> [C:\domains\.com\webapps\malcolment] irectory >>>>>>> [C:\domains\.com\webapps\malcolment] has finished >>>>>>> in [8,469] ms tory >>>>>>> [C:\domains\.com\webapps\notebook] irectory >>>>>>> [C:\domains\.com\webapps\notebook] has finished in >>>>>>> [10,189] ms tory [C:\domains\.com\webapps\order] >>>>>>> irectory [C:\domains\.com\webapps\order] has >>>>>>> finished in [8,501] ms tory >>>>>>> [C:\domains\.com\webapps\payment] irectory >>>>>>> [C:\domains\.com\webapps\payment] has finished in >>>>>>> [14,209] ms tory >>>>>>> [C:\domains\.com\webapps\projectmanager] irectory >>>>>>> [C:\domains\.com\webapps\projectmanager] has >>>>>>> finished in [9,018] ms [C:\Tomcat >>>>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] ptor [C:\Tomcat >>>>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in >>>>>>> [62] ms >>>>>>> >>>>>>> Restart of Tomcat >>>>>>> ------------------------------------------------------------------- >>> - -- >>> tory [C:\domains\.com\webapps\cis] >>>>>>> irectory [C:\domains\.com\webapps\cis] has finished >>>>>>> in [594] ms tory [C:\domains\.com\webapps\gallery] >>>>>>> irectory [C:\domains\.com\webapps\gallery] has >>>>>>> finished in [547] ms tory >>>>>>> [C:\domains\.com\webapps\gl] irectory >>>>>>> [C:\domains\.com\webapps\gl] has finished in [562] >>>>>>> ms tory [C:\domains\.com\webapps\idmanager] >>>>>>> irectory [C:\domains\.com\webapps\idmanager] has >>>>>>> finished in [578] ms tory >>>>>>> [C:\domains\.com\webapps\itemtrack] irectory >>>>>>> [C:\domains\.com\webapps\itemtrack] has finished in >>>>>>> [547] ms tory [C:\domains\.com\webapps\malcolment] >>>>>>> irectory [C:\domains\.com\webapps\malcolment] has >>>>>>> finished in [579] ms tory >>>>>>> [C:\domains\.com\webapps\notebook] irectory >>>>>>> [C:\domains\.com\webapps\notebook] has finished in >>>>>>> [547] ms tory [C:\domains\.com\webapps\order] >>>>>>> irectory [C:\domains\.com\webapps\order] has >>>>>>> finished in [531] ms tory >>>>>>> [C:\domains\.com\webapps\payment] irectory >>>>>>> [C:\domains\.com\webapps\payment] has finished in >>>>>>> [562] ms tory >>>>>>> [C:\domains\.com\webapps\projectmanager] irectory >>>>>>> [C:\domains\.com\webapps\projectmanager] has >>>>>>> finished in [531] ms >>>>>>> >>>>>>> On 5/18/2019 3:18 PM, Rainer Jung wrote: >>>>>>>> Most likely it hangs waiting for enough entropy for random >>>>>>>> number generator seeding. >>>>>>>> >>>>>>>> Try whether the problem goes away if you add >>>>>>>> >>>>>>>> -Djava.security.egd=file:/dev/urandom >>>>>>>> >>>>>>>> to you process flags. If you are using older Java than Java >>>>>>>> 8 (not possible for Tomcat 9 but just in case you also have >>>>>>>> older software stacks running), then it would be >>>>>>>> >>>>>>>> -Djava.security.egd=file:/dev/./urandom >>>>>>>> >>>>>>>> The cryptic /./ is not a typo, you would need it for Java >>>>>>>> before Java 8. >>>>>>>> >>>>>>>> If this doesn't help, then you should try capturing a few >>>>>>>> stack dumps (thread dumps) during the long startup time. >>>>>>>> People here can help interpret them. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm: >>>>>>>>> This is a weird one. It started a few months ago. I >>>>>>>>> have TC 9 running on Windows Server 16. After I reboot >>>>>>>>> the entire server, Tomcat takes forever on startup. It >>>>>>>>> normally starts in about 30 seconds. But after a server >>>>>>>>> reboot it takes up to 15 minutes... chugging along at a >>>>>>>>> snail's pace starting up all of the apps on all of the >>>>>>>>> virtual hosts. It always finally gets there with >>>>>>>>> everything successfully running. Other servers on the >>>>>>>>> same box (Apache, JAMES, ISC BIND, MySQL) don't have any >>>>>>>>> problem starting up. CPU, Disk, Memory, etc. usages are >>>>>>>>> barely showing on the performance graphs. There's nothing >>>>>>>>> in the Catalina log or system.err other than showing a >>>>>>>>> couple of minute gap in the time stamp between each app >>>>>>>>> as it starts up. If I need to reboot again later, it >>>>>>>>> boots again in about 30 sec as expected. >>>>>>>>> >>>>>>>>> First question... any ideas off the top of your head that >>>>>>>>> might be causing this? >>>>>>>>> >>>>>>>>> Second question... is there any other logging I can turn >>>>>>>>> on that might explain what TC is blocking on? >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> Jerry >>>>>>>> ------------------------------------------------------------------ >>> - --- >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>>>>> For additional commands, e-mail: >>>>>>>> users-help@tomcat.apache.org >>>>>>>> >>>>>>> ------------------------------------------------------------------- >>> - -- >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>>>> For additional commands, e-mail: >>>>>>> users-help@tomcat.apache.org >>>>>>> >>>>>> -------------------------------------------------------------------- >>> - - >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>>> For additional commands, e-mail: users-help@tomcat.apache.org >>>>>> >>>>> --------------------------------------------------------------------- >>>>> >>>>> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>>> For additional commands, e-mail: users-help@tomcat.apache.org >>>>> >>>> --------------------------------------------------------------------- >>>> >>>> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>>> For additional commands, e-mail: users-help@tomcat.apache.org >>>> >>> -----BEGIN PGP SIGNATURE----- >>> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ >>> >>> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlzkEWoACgkQHPApP6U8 >>> pFinLRAAttjrmJnG6lwB3pd7kT5IpBk5H/MwJdXvEX8/S/k6MLLgWqWmqQhgRQAZ >>> UeVqJFJFDCrgXymxXNI2f7k3saVA+2Lxafjuft9WQGKs+mTSQR0s82VG6Iko5JzL >>> dZxA5XD+SSU6PafTMQ1hmlVMMDeug+ONqMcUY2XFoZSV6TUk6KqjMiod74We7/b8 >>> 7PSqI4VWSS6fA4/ZoPJNRvGLXtYDu1obVX3YysMb4tSky/5OenJX6rz0mwMenY0z >>> F2p2G2ObNlLKUq7DuAhBcLHB4lbkEv+20Zv6hqMGPlFB8rjiOpUpL86/gfdJY29z >>> KGgWLAw1M4rw2n0GjVihcfYfjVBA931wWwkaXpa6cXCrcyjOcgCH+OlhIqjtoHkW >>> w6mM9m99LqrWRwqqfiHDDxqywdqukfGANwi1QvaPgVkd0+FwGkinh5G3P9au3cLs >>> JKdpGgEog1hJnjkkXWwROlozKiKjMD82V474E4mwjw7nqPI/YXXL5E8VTlhm11JB >>> OWDZ8yNgwA//Aw7JbGdAUPqRSsSqrKIyGYr4voDd6nhs+83R/7z27IyIxgw/kzFU >>> 6xRFOorztK5dOR5TqPZYB67qUL39++fpdayTYDs7f5Y8guajOsWgVTrvWo3bxCkg >>> CxUE7V91tWJYJKlxgbFZyZp4vvSzabf4dq5fxKhPTkYH9pmHhCs= >>> =z1un >>> -----END PGP SIGNATURE----- >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>> For additional commands, e-mail: users-help@tomcat.apache.org >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >> For additional commands, e-mail: users-help@tomcat.apache.org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org