tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phillip Qin <Phillip....@shareowner.com>
Subject RE: Tomcat becoming almost completely non-responsive
Date Mon, 04 Oct 2004 20:08:02 GMT
Can you add the stuff from my email?

-----Original Message-----
From: Kaleb Pederson [mailto:kpederson@mail.ewu.edu] 
Sent: October 4, 2004 12:40 PM
To: Tomcat Users List
Subject: Re: Tomcat becoming almost completely non-responsive


My datasource definition is pretty basic and is as follows:

   <!-- the db object for the mapping database -->
   <Resource name="jdbc/MappingDB" auth="Container" 
type="javax.sql.DataSource" />
   <ResourceParams name="jdbc/MappingDB">
    <parameter>
      <name>removeAbandoned</name>
      <value>true</value>
    </parameter>
    <parameter>
      <name>removeAbandonedTimeout</name>
      <value>60</value>
    </parameter>
    <parameter>
      <name>logAbandoned</name>
      <value>true</value>
    </parameter>
    <parameter>
      <name>factory</name>
      <value>org.apache.commons.dbcp.BasicDataSourceFactory</value>
    </parameter>
    <parameter>
      <name>username</name>
      <value>redirect_user</value>
    </parameter>
    <parameter>
      <name>password</name>
      <value>redirect_pass</value>
    </parameter>
    <parameter>
      <name>driverClassName</name>
      <value>com.mysql.jdbc.jdbc2.optional.MysqlDataSource</value>
    </parameter>
    <parameter>
      <name>url</name>
      <value>jdbc:mysql://myhost/redirects</value>
    </parameter>
    <parameter>
      <name>maxActive</name>
      <value>50</value>
    </parameter>
    <parameter>
      <name>maxIdle</name>
      <value>4</value>
    </parameter>
    <parameter>
      <name>validationQuery</name>
      <value>show tables</value>
    </parameter>
   </ResourceParams>

Thanks for the help.

--Kaleb

On Monday 04 October 2004 8:54 am, Phillip Qin wrote:
> Post your datasource resource definition. You may miss something like
>
>  <parameter>
>   <name>testOnBorrow</name>
>   <value>true</value>
>  </parameter>
>  <parameter>
>   <name>testOnReturn</name>
>   <value>true</value>
>  </parameter>
>  <parameter>
>   <name>minEvictableIdleTimeMillis</name>
>   <value>-1</value>
>  </parameter>
>  <parameter>
>   <name>timeBetweenEvictionRunsMillis</name>
>   <value>300000</value>
>  </parameter>
>  <parameter>
>   <name>numTestsPerEvictionRun</name>
>   <value>1</value>
>  </parameter>
>  <parameter>
>   <name>testWhileIdle</name>
>   <value>true</value>
>  </parameter>
>
>
> -----Original Message-----
> From: Kaleb Pederson [mailto:kpederson@mail.ewu.edu]
> Sent: October 4, 2004 11:19 AM
> To: Tomcat Users List
> Subject: Re: Tomcat becoming almost completely non-responsive
>
>
> John,
>
> Thanks for the feedback.  My code is actually quite simple and I have 
> everything factored out to single function that handles the DB code.  
> I'll paste it below for you, and others to look at.  *Please let me 
> know* if there is an exit path that I somehow missed.
>
> I already have tomcat setup to log abandoned connections.  At your 
> instigation, I grep'd the logs and found a few instances of it in the 
> logs which I was unaware of.  Each one of them corresponds to a time 
> we had high load on the server, so I, hopefully not niavely ;), am 
> guesssing that the abandoned timeout was shorter (currently 60 
> seconds) than the time it took the server to respond to the request 
> (up to 5 minutes).  I guess I should up
>
> the abandoned timeout as that could flame the problem.
>
> If I'm interpreting the thread dump and mysqladmin processlist 
> information correctly, what I'm seeing is that MySQL considers the DB 
> connection to be sleeping, and that 48 of my active db threads were in 
> object.wait() after a call to  AbandonedObjectPool.borrowObject() 
> which seems to get called in every case after a call to 
> DataSource.getConnection():
>
> // note that I removed the org.apache.commons prefix
> 48 like "TP-Processor312" daemon prio=1 tid=0x70331680 nid=0x5e87 
>waiting  for  monitor entry [6df79000..6df7a87c] at
>
dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.jav
>a
>
> :316)
>
>     - waiting to lock <0x7b071438> (a dbcp.PoolableConnectionFactory)
>     at
pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:833)
>     at dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:119)
>     at dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
>     at dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
>     // the function in my filter that checks the DB.
>     at RequestFilterServlet.getUrlMapping(RequestFilterServlet.java:187)
>     ...
>
> Here's the code that handles the db:
>
>  protected RedirectInfo getUrlMapping(String requestURI) {
>
>   RedirectInfo retval = null;
>   Connection conn = null;
>   Statement statement = null;
>   ResultSet result = null;
>
>   try {
>     conn = ds.getConnection();
>    statement = conn.createStatement();
>    statement.setEscapeProcessing(true);
>    result = statement.executeQuery(QUERY_UNIQUE_MATCH + 
> quote(requestURI));
>
>    if (!result.first()) {
>     result = statement.executeQuery(QUERY_CHILD_MATCH_1 + 
> quote(requestURI)
> +
> QUERY_CHILD_MATCH_2);
>    }
>
>    // get the information
>    if (result.first()) {
>     retval = new RedirectInfo();
>     retval.setURL(result.getString(1));
>     retval.setCode(result.getInt(2));
>     debug(requestURI + " -> " + retval.getURL() + " [" +
> Integer.toString(retval.getCode()) + "]");
>    } else {
>     debug("No matching redirect URL found for " + requestURI);
>    }
>   } catch(SQLException e) {
>    log("SQLException caught: " + e.getMessage(), e);
>   } finally {
>    try {
>     // cleanup after myself
>     if (statement != null) statement.close();
>     if (conn != null) conn.close();
>    } catch (SQLException e) {
>     log("Error cleaning up connection/statement: " + e.getMessage(),e);
>    }
>   }
>
>   return retval;
>  }
>
> Thanks for all the help.
>
> --Kaleb
>
> On Sunday 03 October 2004 1:58 pm, jthompson@honda.co.nz wrote:
> > The most likely reason for your connections not being re-used is 
> > that your java code isn't explicitly closing connections, statements 
> > and result sets after use. If visiting all your code is too onerous 
> > a task, you could add the following parameters to your data-source
> > configuration:
> >
> > <parameter>
> >   <name>removeAbandoned</name>
> >   <value>true</value>
> >  </parameter>
> > <parameter>
> >   <name>removeAbandonedTimeout</name>
> >   <value>60</value>
> >  </parameter>
> >
> > The effect of the above is to mark any data-pool connection as 
> > re-usable after 60 seconds of inactivity. You can add the following 
> > to log abandonned connection activity:
> >
> > <parameter>
> >   <name>logAbandoned</name>
> >   <value>true</value>
> >  </parameter>
> >
> > John Thompson
> >
> > |---------+---------------------------->
> > |
> > |         |           Kaleb Pederson   |
> > |         |           <kpederson@mail.e|
> > |         |           wu.edu>          |
> > |         |
> > |         |           02/10/2004 10:07 |
> > |         |           AM               |
> > |         |           Please respond to|
> > |         |           "Tomcat Users    |
> > |         |           List"            |
> > |
> > |---------+---------------------------->
> >
> >---------------------------------------------------------------------
> >---
> >
> >   >--------------------------------------|
> >   >
> >   |       To:       Tomcat Users List <tomcat-user@jakarta.apache.org>
> >   |
> >   |                                      | cc:
> >   |
> >   | Subject:  Tomcat becoming almost completely non-responsive
> >
> >---------------------------------------------------------------------
> >---
> >
> >   >--------------------------------------|
> >
> > Hello,
> >
> > I have an interesting problem.  After a while, tomcat (5.0.27) 
> > becomes almost completely non-responsive.  If I telnet in to port 
> > 8009  (I'm using apache and mod_jk2), I get no response, at least 
> > not within the default timeout. If a browse to a page, I will 
> > generally, after about 4-5 minutes, see a page returned.
> >
> > To narrow down the slowness, I generated a full thread dump, and 
> > found the following information:
> >
> > [ see attachment for more info]
> > Total threads: 180
> > executeQuery: 2 //  executing a db query
> > validateConnection: 0 // trying to validate their connection
> > validateObect: 48 // in 
> > commons.dbcp.PoolableConnectionFactory.validateObject
> > socketAccept: 3 // accepting a socket
> > socketRead0: 10 // reading a socket
> > ReferenceQueue: 1
> > ThreadPool$MonitorRunnable: 2
> > borrowObject and Object.wait: 85 // trying to get an object from the 
> > pool
> > Object.wait: 20 // threads just waiting around
> > Remaining:  9 // misc. threads
> >
> > My database connection is setup so that I have 50 allowed 
> > connections, which matches my 48 in validateObject and 2 executing 
> > queries. However, when I query the database status, I see 2 active 
> > threads and the rest are 'sleeping', just waiting around, as they 
> > would be if the connection pool hadn't released them yet.
> >
> > So, why would there be 48 connections that seemed locked and weren't 
> > querying the DB?  And then the other 85 that were seemingly waiting 
> > on the 45?  Any ideas what might be going on?  The DB is ready?  I 
> > have log abandoned turned and an haven't seen a problem yet.  If the 
> > load drops sufficiently on the server, everything eventually returns 
> > back to normal, otherwise it takes 5-10
> > minutes to get a response from the server.
> >
> > I have attached an abbreviated form of the thread dump which should 
> > provide all the critical information and can provide as much other 
> > information as is necessary.
> >
> > Thanks for the help.  *All* suggestions welcome ;)
> >
> > --Kaleb
> >
> >
> > --------------------------------------------------------------------
> > -
> >  To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
> >
> >
> >
> > --------------------------------------------------------------------
> > -
> > To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org


!DSPAM:41617d66209841547820658!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message