struts-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Balazs Michnay <sztg.sze...@yahoo.com>
Subject Re: [OT] Serious memory leak
Date Tue, 19 Jun 2007 12:22:01 GMT
I forgot to include the link to the image. Here it is:

http://www.inf.u-szeged.hu/~michnay/db/db_connections.jpg

----- Original Message ----
From: Balazs Michnay <sztg.szeged@yahoo.com>
To: Struts Users Mailing List <user@struts.apache.org>
Sent: Tuesday, June 19, 2007 2:16:06 PM
Subject: Re: [OT] Serious memory leak

Thanks for your reply on the memory leak issue
Well, recently I found out that my memory consumption of my application is nothing compared
to the memory consumption of my database server (MySQL).
I'm theoretically using connection pool to save resources of my database server, but each
time I make a query, I have a brand new connection.
Please take a look at the following image that shows the state of my user connection.
I have lots of connections, however theses are only different queries from the same web application.
Why doesn't PID 11 use the connection used by PID 10?
There might be some errors in my code, however, it was previously reviewed and said to be
correct.
Please take a look at the following source...
I have 2 methods related to my connection pool:

-----------------------------------------------GETTING CONNECTION -----------------------------------------------
private Statement getConnectionStatement(Connection conn) throws Exception {
        
        Context ctx = new InitialContext();
        if(ctx == null )
            throw new Exception("Boom - No Context");
        
        Context envCtx = (Context) ctx.lookup("java:comp/env");
        DataSource ds = (DataSource) envCtx.lookup("jdbc/akr_db");
        
        if (ds != null) {
            
            conn = ds.getConnection();
            if(conn == null) throw new Exception();
            
        } else throw new Exception();
        
        return conn.createStatement();
    }
-----------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------CLOSING CONNECTION -----------------------------------------------

private void closeConnection(Connection conn, Statement stmt, ResultSet rst) {
        // Always make sure result sets and statements are closed,
        // and the connection is returned to the pool
        if (stmt != null) {
            try { stmt.close(); } catch (SQLException e) { ; }
            stmt = null;
        }
        if (conn != null) {
            try { conn.close(); } catch (SQLException e) { ; }
            conn = null;
        }
        if (rst != null) {
            try { rst.close(); } catch (SQLException e) { ; }
            rst = null;
        }
    }
-----------------------------------------------------------------------------------------------------------------------------
---------------------------------------USING CONNECTION POOL IN A METHOD---------------------------------

        Connection conn = null;
        Statement stmt = null;  // Or PreparedStatement if needed
        ResultSet rst = null;
        
        String query = "SELECT * FROM MyTable";
        
        try {
            stmt = getConnectionStatement(conn);
            rst = stmt.executeQuery(query);
            
            rst.next();
            
            this.setTartam(rst.getString("tartam"));
            
        } catch (Exception e) {
            System.err.println(e.getMessage());
        } finally {
            closeConnection(conn,stmt,rst);
        }
-----------------------------------------------------------------------------------------------------------------------------

Furthermore, I have a context.xml with the following content:

<Context path="/SZTGKR">
  <ResourceLink global="jdbc/akr_db" name="jdbc/akr_db" type="javax.sql.DataSource"/>
</Context>

-----------------------------------------------------------------------------------------------------------------------------
In the web.xml I have an entry related to my db connection:

 <resource-ref>
        <description> DB Connection Pooling</description>
        <res-ref-name> jdbc/akr_db</res-ref-name>
        <res-type> javax.sql.DataSource</res-type>
        <res-auth> Container</res-auth>
    </resource-ref>   
------------------------------------------------------------------------------------------------------------------------------
<GlobalNamingResources>
    <Environment
      name="simpleValue"
      type="java.lang.Integer"
      value="30"/>
    <Resource
      auth="Container"
      name="jdbc/akr_db"
      type="javax.sql.DataSource"
      maxActive="100"
      maxIdle="30"
      username="balazs"
      maxWait="10000"
      driverClassName="com.mysql.jdbc.Driver"
      removeAbandoned="true"
      password="12345"
      url="jdbc:mysql://localhost:3306/akr_db"
      removeAbandonedTimeout="30"/>
    <Resource
      auth="Container"
      description="User database that can be updated and saved"
      name="UserDatabase"
      type="org.apache.catalina.UserDatabase"
      pathname="conf/tomcat-users.xml"
      factory="org.apache.catalina.users.MemoryUserDatabaseFactory"/>
  </GlobalNamingResources>
------------------------------------------------------------------------------------------------------------------------------

That's all I have related to database connection.
What am I doing wrong?

Thanks a lot,

MB



----- Original Message ----
From: Christopher Schultz <chris@christopherschultz.net>
To: Struts Users Mailing List <user@struts.apache.org>
Sent: Wednesday, May 30, 2007 5:45:11 PM
Subject: Re: [OT] Serious memory leak

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Balazs,

I just realized that we're talking about approximately 1MB of char[]
data... what's the big deal about that? ;)

Balazs Michnay wrote:
>> Can you expand the "+" next to Arrays.copyOfRange to see what the rest
>> of the stack trace is? I'm sure that a lot of things call
>> Arrays.copyOfRange, so it's hard to determine what the problem is. Post
>> another screenshot if you can.
> 
> Sure,
>  here it is (profiler3_expanded.JPG):
> http://www.inf.u-szeged.hu/~michnay/profiler/
> As you can see, now there are other methods as well allocating char[] objects.
> It's pretty confusing and very hard to track how exactly my pages are processed by the
server... at least, it is for me... :)

It looks like most of those char[] objects are being allocated by Sun's
HTTP client. What are you loading over HTTP? Class files? Apparently,
you have some big arrays lying around:

 2 * 16k bytes allocated by HttpClient.parseHTTP
39 * 16k bytes allocated by HttpClient.<init> (the constructor)

These look like buffers. I'd be interested in seeing what is using
HttpClient in your code (or Struts, I suppose).

The other section (copyOfRange) has created lots of String objects. Some
of them are obvious: you are reading properties from properties files
and they take up memory. StringBuilder is often used by the compiler to
assemble Strings combined using the + operator (for Strings). You can
use it yourself, too, just like a StringBuffer. Somewhere, strings are
being created (which, of course, are made-up almost entirely of char[]
objects).

> I'll
> try to factor out the JNDI parts. Should I factor out the creation of
> statement and result objects as well, or only the connection should be
> factored out?

I would only do the Connection. What else would you do? Have a method
like this?

public Statement createStatement(Connection conn, String sql)
{
    return conn.createStatement(sql);
}

??! That's a pretty worthless method.

> Should I create another method to free up the resources?

I have done that myself:

protected void close(Connection conn, Statement s, ResultSet r)
{
 /// this guy is protected because it's in a superclass

 // for each object, check for null, try { close } catch { log }
}

>> In terms of design, I would write that method to return a List of
>> application-specific objects instead of maintaining two separate
>> (untyped) lists that just happen to be related. It is a much more
>> understandable design and yields much more readable code. It will also
>> simplify your JSP loop, etc.
> 
> What do you mean by "List of application-specific objects"? Is it a
> multi-dimensional ArrayList object? I'm very interested in creating
> well-readable and well-understandable code, so some samples would be
> appreciated :) (sorry, I'm not very experienced in Java)

I know how you feel. We used to have a programmer who didn't realize he
could create his own types. Stop thinking that everything has to be a
String hidden inside a baffling maze of nested Lists and Maps. If you
have an entity that you like to think of as a "person" (for example),
then you can represent this data in at least two ways:

1. Use a List of Strings where the index of the string has meaning,
   but only in your head. Something like this:

   ArrayList person = new ArrayList();
   person.add("Chris");   // 0 == first name
   person.add("Schultz"); // 1 == last name
   person.add("Male");    // 2 == gender

   ArrayList people = new ArrayList();
   people.add(person);
   // etc.

2. Use an object you define yourself.

   public class Person
   {
      String firstName;
      String lastName;
   }

   ArrayList people = new ArrayList();
   people.add(new Person("Chris", "Schultz", "Male"));
   // etc.

If you use solution #1, you'll never know if the ArrayList you're
getting is an array of String representing "person" entities or a list
of strings representing "Account" entities, or whatever. You basically
have to cross your fingers and hope that the documentation for the
method is correct.

If you use #2, you can use the compiler and runtime to enforce your
notion of what is what: That list /does/ contain Person objects. You
seriously simplify your code and rarely get confused. You save a lot of
memory, too, since ArrayLists have overhead that you don't need if you
use your own objects.

These things are typically called "business objects", but only when they
map into domain-specific entities (meaning that they represent real
concepts in the language of the problem... like "Account" for financial
software, or "PurchaseOrder" for an ordering system).

You are free to create other types that are useful to you even if they
are not "business objects" per se.... any time you find yourself
stuffing things like bare Strings into Maps or Lists, ask yourself if
you shouldn't be defining your own type (or class... or object... or
whatever you want to call it).

>> Do you see any place where objects could not be freed-up? Where?
> 
> I'll show you. Please take a look at
>  profiler4.JPG found at the same URL.
> The big black curly bracket shows how the memory
> is eaten up by simple page reloads. The profiler's GC button can be
> found at the upper-left corner indicated by the green arrow. By
> pressing GC, a small amount of memory is freed up, which can be seen in
> the yellow ellipse. It's a little slope indicating that a very little
> amount of memory could be freed up, no matter how many times I press
> the GC button. What about the rest? That's pretty confusing and
> sometimes a bit annoying, because I don't see where my code wastes ~25
> Mbytes of memory...

Most of that memory is taken up by things other than char[] objects.
Look again at your old screenshots.... there's only about 1MB being used
by char[] arrays.

> Yes,
> you're absolutely right, that's exactly what I think. I was about to
> accept that I'm just simply unable to comprehend what happen behind the
> scenes, but now - thanks to your helpfulness and willingness to help -
> things are getting clearer...

It's pretty complicated. Java's GC is pretty good. Java also takes up a
lot of memory with runtime objects, etc. and it's suprising to see that
sometimes. Unless you are actually observing a memory problem, you
probably don't have one.

This is further complicated by the fact that there are different
generations of memory... objects that live long enough are promoted to
older generations as seen by the GC. Eventually, an object will be
considered permanent and pretty much never be cleaned up.

It's possible that what you're done is get yourself into a steady-state
situation where there's just no more memory to free after startup and a
few page re-loads. A 30MB heap is perfectly reasonable for a running
application server.

> Apart from the fact that my code is whether wastes or doesn't
> waste ~25 Mbytes of memory by virtually "doing nothing", it's very
> calming to see that other factors not directly related to my code can
> affect memory load.

There's a /lot/ of stuff you don't know about under the covers in the
app server (let alone the JVM). The application server itself needs a
lot of configuration objects lying around to operate properly. It needs
to have the JSP compiler loaded so it can compile your JSPs. It has to
have its own classes for application, session, request/response, and
page management loaded.

Each request creates (and usually destroys) tons of objects (request,
response, sometimes multiple wrappers for each, session, request
parameters (strings!) and headers (more strings!), event objects for
lots of things, logging events, etc.) that you never really have to
think about.

If you have JMX enabled, nearly everything that occurs generates an
event that it broadcast to listeners that may react by creating more and
more objects.

You just have to understand that your code does not run in a vacuum. You
may have written a single JSP but there are miles (kilometers?) of code
behind your ability to type in a few magic words into a .jsp file and
have it generate HTML when a client requests that page. It's
mind-boggling, actually.

My advice is not to panic until you start getting OutOfMemoryErrors
thrown from your application. Java's defaults are pretty good memory
settings. If you start to have problems with those, then it's time to
see if you have some kind of leak.

I'd also get more familiar with Java in general. After knowing a ton
about Java, I remember leafing through Bruce Eckel's "Thinking in Java"
and thinking that it might be quite good for a Java newcomer. It's even
available for free online: http://www.mindview.net/Books/TIJ/

You might want to check it out. Hopefully, it will give you the
background you'll need to better understand the issues facing you, now.

Good luck,
- -chris

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGXZwH9CaO5/Lv0PARAhLMAJ90nJZaP4HCXA9hS2tCncvAwXK4fQCgmb9D
xa+7lH+9gWkHE9GYfwpQ9ro=
=Clx8
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
For additional commands, e-mail: user-help@struts.apache.org








 
____________________________________________________________________________________
No need to miss a message. Get email on-the-go 
with Yahoo! Mail for Mobile. Get started.
http://mobile.yahoo.com/mail 






       
____________________________________________________________________________________
Moody friends. Drama queens. Your life? Nope! - their life, your story. Play Sims Stories
at Yahoo! Games.
http://sims.yahoo.com/  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message