tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Schultz <ch...@christopherschultz.net>
Subject Re: Tomcat: client disconnects?
Date Thu, 15 Jan 2009 21:58:20 GMT
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

SP,

pichels wrote:
> Christopher Schultz-2 wrote:
>> Is this a browser error message, or something coming from the Tomcat side?
> 
> Browser message - "Page cannot be displayed" - in Firefox or IE6/7 - even
> using the IP address bypassing DNS of the Web server.

Hmm. I wonder if you are actually having trouble contacting the server
- -- that is, not an application error but possibly run out of available
queued sockets on the server or some odd network error. Maybe check
using a packet capture like Wireshark to see if the server is even
picking up the phone.

>> Are you using a connection pool? If so, what is the configuration? If
>> not, how are you connecting to the database (and is there a limit on the
>> number of db connections you allow)?
> 
> No - Pooling. JDBC connector. No limit.

That's bad. When you get a storm of activity, you can choke both your
app server /and/ your database server if the number of connections is
not limited. MySQL is pretty fast to establish a new connection, but
many databases take significant amount of time to create new connections
and so pooling is also a performance optimization.

> Yes, for Mediawiki we are setup to use index.php - understood. 
> With Tomcat we don't use index.jsp as a front-controller for servlet
> traffic.

Okay. Does index.jsp do anything significant, then? It sounded like you
were saying that you observe many problems with that specific page.
Maybe you are just having generic app server slowness and since the home
page is popular, you see a lot of failures there. Hmm.

>> Would you care to post some of those? What are the most popular
>> exceptions? What seem to be the worst-sounding?
> 
> Most Popular below:
> 
> WARNING: Error sending end packet 
> java.net.SocketException: Broken pipe 
>         at java.net.SocketOutputStream.socketWrite0(Native Method) 

This is happening because the client hung up the phone before reading
the entire response (or any of it). This will happen if a client hits
the STOP button or clicks on another link or hits RELOAD while an
existing request is still pending. Users will do this if the site feels
too slow, so it's a vicious cycle: slow site generates additional
requests from anxious users which ... slows down the site. :(

> Mostly coming from Tomcat/webapp. Yes, one we fixed but we are still
> having issues w/ slow queries.
> Ok, we'll look into running MySQL Explain.

You might want to get the O'Reilly book on optimizing MySQL. It's thin
(but cheap!) and gives you some good information about what to look for.
Unfortunately, MySQL isn't really that tunable. On the other hand, MySQL
requires very little in the way of tuning!

>>>> We have also tried using WinXP host files and using an IP address to
>>>> bypass
>>>> DNS altogether - no luck.
> Are you actually performing DNS lookups?
> 
> Yes, with all the other users/clients.
> We have a pilot "test" group of users only using the IP address in the URL
> box of their browsers.

That shouldn't matter: the client's computer will do DNS resolution
before your server is even contacted. If there are server-related DNS
issues, it will be because the server is trying to do reverse-lookups on
the clients, not because of what the client's URL bar shows.

>>>> Problem around 4:15PM - Jan 9, 2009:
>>>> http://10.97.24.23/Sales/IndentedATPServlet?partNum=086EXUCCCXMA&qty=1&warehouse=M
> What was the problem?
> 
> User was viewing a Servlet page that was querying the AS400 DB2 DB on the
> backend for a part number.
> Tomcat may have been waiting for a long time and then teh user recieved
> "page cannot be displayed" or "Server cannot be found" in IE7 browser.

Hmm. MySQL or DB2? How many databases are you using?

>  http://www.nabble.com/file/p21485970/dns.jpeg

OIh, that could happen for lots of reasons, including "the server just
isn't running at all". I hate those "helpful" error messages. Wild
goosechases all around.

> NOTE: The connection is thru Tomcat JDBC connector to AS400 DB2 DB.

Technically, the JDBC connector is not Tomcat-specific.

> We have Apache & Tomcat Debugging set very high. And have MySQL slow
> queries and error logs.

Note that debugging logging can slow down your system a lot. File IO is
much worse than, say, object creations and traversals.

> Can't find any relevant to error. 
> Can we set Tomcat debugging with date/time stamps or will Log4j help more
> with debugging?

Heh. See this week's archives for log4j issues people have been having.
I wouldn't worry about that just yet.

>> This is why I was asking about connection pooling: it looks like Tomcat
>> is murdering your DB. This could be due to too many connections to
>> MySQL, overly high query volume, or sloppy code in your application
>> leaking database resources.
> 
> Java Developer is looking into code constantly and cleaning up code.
> He agreed that there is old sloppy code perhaps and possibly too many
> connections.
> Can we control the connections or help diagnose code errors in an easier
> way? 

The very best thing to do is to use a database connection pool. Tomcat
supplies one, which is convenient. What is inconvenient is that you
probably will have to change the way you get new database connections in
your code. Hopefully, your application uses a single method to get a
connection, and that method can simply be re-implemented to use a
connection pool. Otherwise, you have a lot of work ahead of you.

Tomcat's connection pool can also be put into a debugging mode where
connections improperly closed (or not at all) can be logged, including a
stack trace for where those connections were requested. It's a great
help when attempting to locate and fix these kinds of issues.

I suspect that switching to use a connection pool will solve around 50%
of your problems, and then give you a lot of information about how to
further improve your application. You may find that your application
leaks connections all over the place which can cause both Tomcat and
MySQL to thrash, which makes the user experience suck for the user. As
you mentioned, bouncing Tomcat makes everything better, so it sounds
like your application is a bit sloppy when it comes to resource management.

At least you're not running out of memory ;)

It's also good that you have an engineer who is willing to go after old
code and fix it rather than just saying "I didn't write it, don't blame me".

If you're ready to make the jump to a connection pool (do it!), let me
know and I'll give you info on configuration and how to do the code
(actually, it's all here in the Tomcat site:
http://tomcat.apache.org/tomcat-5.5-doc/jndi-datasource-examples-howto.html).


A few caveats about the above page:

- - The instructions tell you to put the <Resource> element into the
server.xml file inside your <Context>. Assuming you are using a
META-INF/context.xml file (you should be!) then put this configuration
into your context.xml file, not into server.xml.

- - The instructions tell you to use the autoReconnect=true parameter in
your JDBC connection URL. You should not (MySQL has asked users not to
use this for, literally, years). Instead, set the validationQuery
attribute of the <Resource> element to something like "/* ping */ SELECT
1". This will issue a simply query to the server before the connection
is handed over to your application, so you'll always get a fresh one.

- - The MySQL example really contains some MySQL-specific stuff, then some
generic configuration (using the MySQL driver as an example) and then
stops. The Oracle example actually contains the code to get the
DataSource (and then the JDBC connection) from the server. This is the
same no matter what backend database you have.

Finally, note that you can have multiple DataSources configured. Feel
free to configure one for MySQL and another for your DB2 instance.

> I've sent the logs to RHEL(RedHat Linux) support and they have been
> reviewing and think we have a network issue - router or VMware problems
> since this is a VMware guest.

Well, vmware guests can certainly have issues. You aren't using one for
production, are you? I'm not sure I would recommend that.

> I was also told by another tech at RHEL support that RHEL5 runs fine on
> WMWaare as is if it were a physical box - confusing.

It should run fine. I use vmware guests all the time for testing and
I've never had any problems. But, I don't use them for load testing or
anything like that -- just easy ways to lug a linux server around on my
win32 laptop.

> I see there are many ways to load test" MySQL - do you have any
> suggestions on where I can start?
> A free/Open source tool would be preferred. <smile>

MySQL is probably running very well. As I said earlier, MySQL has very
few performance-tuning settings -- mostly changing the sizes of various
caches. You'll get much more of a performance boost our of properly
sizing resource pools (such as db connections) in your application and
making sure that you properly clean up after yourself.

Let us know what you decide, and how things turn out. Feel free to post
back with more info and/or questions.

Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklvsVUACgkQ9CaO5/Lv0PArfwCgqnEZgWX3qrCAlbMyYJ2kNZVG
6jwAoKAAyITQV62uh2hvopKlZR8Mc5rQ
=B62W
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message