commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Mastroianni" <MMastroia...@choicestream.com>
Subject HttpClient -- possible resource leak?
Date Wed, 09 Jun 2004 14:43:10 GMT
I have a multi-threaded app, using Httpclient to download a few thousand urls at a time. Currently,
I have one MultiThreadedHttpConnectionManager, which the thread manager creates, and passes
around to each of its worker threads.

Each thread has a queue of urls, and it creates a new HttpClient, using the ConnectionManager,
for each one. I've tried using one, created at construction time for each worker thread, and
gotten no luck.

The worker threads make executeMethod calls, and I notice that I'm leaking a lot of memory
(it looks like the memory usage goes up every time I successfully download a page). It seems
as if perhaps the underlying buffer of the GetMethod is not being cleaned up. I'm calling
release on the GetMethod in a finally block. A relevant piece of code is below:

            private void SpiderUrlImpl()
            {
                        HttpMethod method = new GetMethod(m_sUrl);
                        try
                        {
                                    //if(m_State == null)
                                    //{
                                               
m_State = new HttpState();
                                               
m_State.setCookiePolicy(CookiePolicy.RFC2109);
                                    //}
            
                                    m_client.setState(m_State);
                                    m_client.setConnectionTimeout(m_timeout);

                                    method.setFollowRedirects(true);
                                    method.setStrictMode(false);
                                    String responseBody
= null;
                                    
                                    int iCode    = m_client.executeMethod(method);
                                    responseBody = method.getResponseBodyAsString();
                                    Header hLoc  = method.getResponseHeader("Location");
                                    
                                    java.io.FileWriter
fw = new java.io.FileWriter(m_sPath + "\\" + m_sFile);
                                    fw.write(responseBody);
                                    w.close();
                        }//TODO: LOG STUFF GOES HERE
                        catch (org.apache.commons.httpclient.HttpException
he)
                        {
                            System.err.println("Http error connecting
to '" + m_sUrl + "'");
                            System.err.println(he.getMessage());
                        }
                        catch (IOException ioe)
                        {
                            System.err.println("Unable to connect
to '" + m_sUrl + "' or print file + '" +  m_sPath + "\\" + m_sFile + "'");
                            System.err.println(ioe.getMessage());
                        }
                        catch(Exception eExc)
                        {
                            System.err.println(eExc.getMessage());
                        }
                        finally
                        {
                            method.releaseConnection();
                        }
            }
}



---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org


Mime
View raw message