Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Tomcat Users List" <users@tomcat.apache.org>
Received-SPF: pass (athena.apache.org: local policy)
From: Debbie Shapiro <DShapiro@cardiacscience.com>
To: Tomcat Users List <users@tomcat.apache.org>
Date: Mon, 12 Mar 2012 10:48:21 -0700
Subject: RE: Web service not starting up as expected
Thread-Topic: Web service not starting up as expected
Thread-Index: Ac0AdkhKjru/2K8DSFSRjG8GU3r85wAAVzng
Message-ID: <30E219DE264470478E86828191E9B7510553D9B672@csmail>
References: <30E219DE264470478E86828191E9B7510553D9B3B1@csmail>
 <4F5E1F22.9010907@christopherschultz.net>
In-Reply-To: <4F5E1F22.9010907@christopherschultz.net>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]=20
Sent: Monday, March 12, 2012 9:07 AM
To: Tomcat Users List
Subject: Re: Web service not starting up as expected


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Debbie,

On 3/9/12 3:57 PM, Debbie Shapiro wrote:
> I'm pretty sure that's accurate in this case, however. I had
> 7.0.23 installed as of last week and then this week did the upgrade
> to 7.0.26. I'm wondering about your question regarding timing
> though. It's possible that Tomcat is doing more cleanup than it has
> in the past and it's taking longer to shut down? Maybe I do just
> need some kind of pause in between stopping and restarting.

Perhaps.

> I either directly run the .bat command or launch the scheduled
> task that normally runs the same .bat command.
>=20
> The contents of the .bat file is as follows:
>=20
> sc \\bidev2 stop Tomcat7
>=20
> taskkill /F /FI "IMAGENAME eq java.exe" /FI "USERNAME eq NT
> AUTHORITY\SYSTEM"

So there's no delay between requesting a service-stop and calling
taskkill (I assume that's like *NIX 'kill' that basically murders the
process)? I would consider giving Tomcat some time to shut down. You
never know what kinds of things need time to shut down.

> REM delay batch job to wait for services to stop

I don't see an actual delay, do you?

No, I had left that there as a placeholder from my previous code. When I se=
t up this server as 64-bit, I was trying to hold off from installing the Mi=
crosoft resource toolkit that contains the sleep command. I have that insta=
lled now and added a sleep command here.

> sc \\bidev2 start Tomcat7

Yeah, I'll bet you need some delay in this script in order for it to
work properly.

> I have actually mentioned the memory leaks to them before. They
> said not to be concerned, but maybe others will report this as
> well.

They aren't concerned because they aren't running their own software
in production :( They need to fix this because it is definitely a
memory leak.

I agree, this doesn't look right. I'll ping them about it again.


>> If this is really the end of the log file, then it looks like
>> Tomcat didn't even attempt to start itself again. When you
>> perform a restart, do you move the log file out of the way or
>> anything like that? If so, is there a "new" log file that might
>> have some indication on an error?
>=20
> This was the end of the file, yes. I thought it looked odd too. I=20
> don't move any of the Tomcat logs. I only compress and rename the
> old InetSoft logs.

Okay, then Tomcat isn't even trying to start again (or at least isn't
logging anything). It's possible that if the old process is still
shutting down, the new process can't get a lock on the log file for
writing, so no logs are written.

> I have a scheduled task that kicks off the .bat file I copied
> above.

Okay, add some delay. I dunno how to do that on win32.

Install the Microsoft Resource Toolkit. It contains a sleep command.

> This second scheduled task started up fine. So maybe it is just a=20
> timing issue. 7.0.26 takes longer to shut down than 7.0.23 maybe?

It probably takes about the same amount of time, you're just noticing
it now for some reason.


Something must be different because 7.0.23 didn't require me to add the del=
ay to my script and I hadn't received this error until I installed 7.0.26. =
The sleep command seems to be working. I have to add at least 30 seconds or=
 it doesn't startup.

> I'm going to test out a delay in the restart after I shut down,
> but just wanted to throw out there that after my stop service
> command, which happens at exactly 11:30, the restart failure in the
> event viewer has a timestamp of 11:30:06. That's not much of a
> delay between the stop and restart, so maybe it's still
> technically shutting down at this point.

6 seconds to shut down wouldn't be out of the ordinary. If a webapp is
doing things properly, it will have some of its own shutdown code
running to clean up resources, etc. that will take non-zero time. Most
of the shutdown time will be the webapp and not Tomcat itself, which
really just has to un-deploy the webapp and then stop the connectors.

> Yes, it is pretty common in the windows world to restart servers
> and services due to various memory issues. I'm doing this because
> the application is a reporting tool and if the users run any
> enormous reports that cause it to run out of java heap or
> something, this provides for an automated way of maintaining the
> application so that it doesn't require manual intervention to fix
> it and I know that even if no one calls me, it will be reset at
> some point. I also use it as an opportunity to save off a daily
> copy of the InetSoft logs if I need to go back and review errors
> later. Personally, I'm a Mac fan at heart. :-)

I'm curious how long you could run without a restart. Usually for
reporting types of webapps, it's either a single request that
brings-down the server (e.g. it needs a huge amount of memory to run a
single report) or you have lots of smaller requests that need a ton of
memory in aggregate.

It's not really possible to fix the former, other than getting more
memory or changing the way reports are done to use less of it. Threads
can't get their "own" memory usage so you can't really limit one
particular request.

It's easy to fix the latter: just find our whatever you maximum load
is in terms of simultaneous reports, and simply never let that many
run at once. You can do that with a report-counter in the webapp
itself, or you can do it by limiting the maximum number of connections
that the server will accept.

For my money, I'd off-load the actual report-generation to an
out-of-process (duh) process and pick-up the result when it's done.
That way, your webapp doesn't become unstable even if one report runs
out of memory.


Yes, in our situation, it's usually someone running a HUGE report that take=
s the server down. Your last paragraph, how would one off-load the report-g=
eneration? Is it possible to do this with a third party application? I don'=
t have access to their code. I'm just implementing their solution.


- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9eHyIACgkQ9CaO5/Lv0PD/5gCeLxNUkb4LBZgU77rNx5rGng1u
fdIAoIOhNfgFatg8ukVkAZTiZyH0NUsl
=3Dqjf7
-----END PGP SIGNATURE-----


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org