From users-return-267557-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Mon May 20 08:44:17 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id CD704180627 for ; Mon, 20 May 2019 10:44:16 +0200 (CEST) Received: (qmail 92874 invoked by uid 500); 20 May 2019 08:44:13 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 92863 invoked by uid 99); 20 May 2019 08:44:13 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 May 2019 08:44:13 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1241DC09BB for ; Mon, 20 May 2019 08:44:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.8 X-Spam-Level: X-Spam-Status: No, score=0.8 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id bjec6F2T-pOo for ; Mon, 20 May 2019 08:44:10 +0000 (UTC) Received: from thor.wissensbank.com (thor.wissensbank.com [81.169.250.120]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 46FC55FB15 for ; Mon, 20 May 2019 08:44:08 +0000 (UTC) Received: from thor.wissensbank.com (localhost [127.0.0.1]) by thor.wissensbank.com (Postfix) with ESMTP id AC0A81E100155 for ; Mon, 20 May 2019 10:44:02 +0200 (CEST) Received: by thor.wissensbank.com (Postfix, from userid 500) id 9FBF61E100158; Mon, 20 May 2019 10:44:02 +0200 (CEST) Received: from [192.168.245.152] (unknown [188.130.198.139]) (Authenticated sender: andre.warnier@ice-sa.com) by thor.wissensbank.com (Postfix) with ESMTPA id EE0AD1E100155 for ; Mon, 20 May 2019 10:44:01 +0200 (CEST) Subject: Re: Massive Startup Time after Server Reboot To: users@tomcat.apache.org References: <761e3fa1-6d82-9f36-9edd-9973af85285d@malcolms.com> <8ab39715-ebda-d743-5aaa-d57c04d1debf@malcolms.com> From: =?UTF-8?Q?Andr=c3=a9_Warnier_=28tomcat=29?= Message-ID: <5CE268CD.3030907@ice-sa.com> Date: Mon, 20 May 2019 10:43:57 +0200 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <8ab39715-ebda-d743-5aaa-d57c04d1debf@malcolms.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP On 20.05.2019 00:09, Jerry Malcolm wrote: > Just an FYI.... this server has been in production a little over a year. This TC load > problem only started 2-3 months ago. So 'something' changed to cause this problem to > manifest itself. It's been too long to try to correlate a specific server change to the > start of this problem. But I just wanted to point out that this problem has NOT been > around since day 1 on this server. Just in case, since this is a Windows server .. 2-3 months ago, an update of a virus scanner ? (of the intrusive kind which pre-checks each file that wants to be opened) Checking may be easy or not, depending on your access : disable the virus scanner just the time to start tomcat and check. > > Jerry > > On 5/19/2019 5:03 PM, Jerry Malcolm wrote: >> Rainer, >> >> No change with the urandom parm. I am attaching a portion of the Catalina log. The >> first half shows between 8 and 15 seconds to deploy each app for a single virtual host >> (there are no war files, the app is already exploded in the appbase dir). I have quite >> a few virtual hosts with several apps each. That initial server start took 21+ minutes. >> I then restarted the TC service and got the ~.5 sec start per webapp shown in the 2nd >> half of the log below. >> >> Can you refresh me on how to capture the stack dumps you suggested? It's been a while.... >> >> Thx. >> >> Jerry >> >> First start of Tomcat after server reboot >> --------------------------------------------------------------------- >> [C:\domains\.com\webapps\JSPWiki.war] has finished in [8,579] ms >> tory [C:\domains\.com\webapps\cis] >> irectory [C:\domains\.com\webapps\cis] has finished in [11,486] ms >> tory [C:\domains\.com\webapps\gallery] >> irectory [C:\domains\.com\webapps\gallery] has finished in [9,204] ms >> tory [C:\domains\.com\webapps\gl] >> irectory [C:\domains\.com\webapps\gl] has finished in [8,469] ms >> tory [C:\domains\.com\webapps\idmanager] >> irectory [C:\domains\.com\webapps\idmanager] has finished in [8,689] ms >> tory [C:\domains\.com\webapps\itemtrack] >> irectory [C:\domains\.com\webapps\itemtrack] has finished in [6,907] ms >> tory [C:\domains\.com\webapps\malcolment] >> irectory [C:\domains\.com\webapps\malcolment] has finished in [8,469] ms >> tory [C:\domains\.com\webapps\notebook] >> irectory [C:\domains\.com\webapps\notebook] has finished in [10,189] ms >> tory [C:\domains\.com\webapps\order] >> irectory [C:\domains\.com\webapps\order] has finished in [8,501] ms >> tory [C:\domains\.com\webapps\payment] >> irectory [C:\domains\.com\webapps\payment] has finished in [14,209] ms >> tory [C:\domains\.com\webapps\projectmanager] >> irectory [C:\domains\.com\webapps\projectmanager] has finished in [9,018] ms >> [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] >> ptor [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in [62] ms >> >> Restart of Tomcat --------------------------------------------------------------------- >> tory [C:\domains\.com\webapps\cis] >> irectory [C:\domains\.com\webapps\cis] has finished in [594] ms >> tory [C:\domains\.com\webapps\gallery] >> irectory [C:\domains\.com\webapps\gallery] has finished in [547] ms >> tory [C:\domains\.com\webapps\gl] >> irectory [C:\domains\.com\webapps\gl] has finished in [562] ms >> tory [C:\domains\.com\webapps\idmanager] >> irectory [C:\domains\.com\webapps\idmanager] has finished in [578] ms >> tory [C:\domains\.com\webapps\itemtrack] >> irectory [C:\domains\.com\webapps\itemtrack] has finished in [547] ms >> tory [C:\domains\.com\webapps\malcolment] >> irectory [C:\domains\.com\webapps\malcolment] has finished in [579] ms >> tory [C:\domains\.com\webapps\notebook] >> irectory [C:\domains\.com\webapps\notebook] has finished in [547] ms >> tory [C:\domains\.com\webapps\order] >> irectory [C:\domains\.com\webapps\order] has finished in [531] ms >> tory [C:\domains\.com\webapps\payment] >> irectory [C:\domains\.com\webapps\payment] has finished in [562] ms >> tory [C:\domains\.com\webapps\projectmanager] >> irectory [C:\domains\.com\webapps\projectmanager] has finished in [531] ms >> >> On 5/18/2019 3:18 PM, Rainer Jung wrote: >>> Most likely it hangs waiting for enough entropy for random number generator seeding. >>> >>> Try whether the problem goes away if you add >>> >>> -Djava.security.egd=file:/dev/urandom >>> >>> to you process flags. If you are using older Java than Java 8 (not possible for Tomcat >>> 9 but just in case you also have older software stacks running), then it would be >>> >>> -Djava.security.egd=file:/dev/./urandom >>> >>> The cryptic /./ is not a typo, you would need it for Java before Java 8. >>> >>> If this doesn't help, then you should try capturing a few stack dumps (thread dumps) >>> during the long startup time. People here can help interpret them. >>> >>> Regards, >>> >>> Rainer >>> >>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm: >>>> This is a weird one. It started a few months ago. I have TC 9 running on Windows >>>> Server 16. After I reboot the entire server, Tomcat takes forever on startup. It >>>> normally starts in about 30 seconds. But after a server reboot it takes up to 15 >>>> minutes... chugging along at a snail's pace starting up all of the apps on all of the >>>> virtual hosts. It always finally gets there with everything successfully running. >>>> Other servers on the same box (Apache, JAMES, ISC BIND, MySQL) don't have any problem >>>> starting up. CPU, Disk, Memory, etc. usages are barely showing on the performance >>>> graphs. There's nothing in the Catalina log or system.err other than showing a couple >>>> of minute gap in the time stamp between each app as it starts up. If I need to reboot >>>> again later, it boots again in about 30 sec as expected. >>>> >>>> First question... any ideas off the top of your head that might be causing this? >>>> >>>> Second question... is there any other logging I can turn on that might explain what TC >>>> is blocking on? >>>> >>>> Thanks. >>>> >>>> Jerry >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >>> For additional commands, e-mail: users-help@tomcat.apache.org >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org >> For additional commands, e-mail: users-help@tomcat.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org > For additional commands, e-mail: users-help@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org