commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikael Andersson" <mail.mi...@gmail.com>
Subject [httpclient] how to properly set the timeout + more
Date Fri, 19 May 2006 14:37:13 GMT
Hi,
I am running into some issues when using the HttpClient to retrieve quite
large datasets (20-30) MB. I am thinking that it may have something to do
with timeout settings.

When accessing a resource which does a fair bit of processing the data
before returning it, the transfer fails after some 13MB. But when accessing
the same data but specifying that I want the raw data which it streams back
faster, I can retrieve the whole data set.

I am retrieving data which contains several pieces of data in one go, and I
know that data streamed back comes in chunks. The remote resource takes the
first piece of data and annotates it with html and returns it, and then the
next and so on.

The mimetype for the HTML annotated data is also text/html, for the raw data
it is text/plain.

The way I specify the time out now is like this:
private static long TIME_OUT = 30000;

MultiThreadedHttpConnectionManager connManager = new
MultiThreadedHttpConnectionManager();
HttpClientParams httpClientParams = new HttpClientParams();
httpClientParams.setConnectionManagerTimeout( TIME_OUT );
httpClient = new HttpClient( httpClientParams, connManager );

[......]

HttpMethodParams methodParams = new HttpMethodParams();
methodParams.setSoTimeout( (int)TIME_OUT );
getMethod = new GetMethod( urlStr );
getMethod.setParams( methodParams );
getMethod.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
                                           new
DefaultHttpMethodRetryHandler(3, false));

I get the following Exception:
java.lang.ArrayIndexOutOfBoundsException: 3000
        at org.mortbay.util.ByteArrayOutputStream2.writeUnchecked(
ByteArrayOutputStream2.java:41)
        at org.mortbay.jetty.HttpGenerator$OutputWriter.write(
HttpGenerator.java:1189)
        at java.io.PrintWriter.write(PrintWriter.java:384)
        at java.io.PrintWriter.write(PrintWriter.java:401)
        at uk.ac.ebi.jdbfetch.BatchFetcher.fetchData(BatchFetcher.java:90)
        at
uk.ac.ebi.jdbfetch.servlet.commands.FetchAndDisplayCommand.fetchAndDisplay(
FetchAndDisplayCommand.java:115)
        at
uk.ac.ebi.jdbfetch.servlet.commands.FetchAndDisplayCommand.execute(
FetchAndDisplayCommand.java:56)
        at uk.ac.ebi.jdbfetch.servlet.JDBFetchServlet2.processRequest(
JDBFetchServlet2.java:137)
        at uk.ac.ebi.jdbfetch.servlet.JDBFetchServlet2.doGet(
JDBFetchServlet2.java:72)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:747)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java
:423)
        at org.mortbay.jetty.servlet.ServletHandler.handle(
ServletHandler.java:355)
        at org.mortbay.jetty.servlet.SessionHandler.handle(
SessionHandler.java:219)
        at org.mortbay.jetty.handler.ContextHandler.handle(
ContextHandler.java:540)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:145)
        at org.mortbay.jetty.handler.HandlerCollection.handle(
HandlerCollection.java:128)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(
HandlerWrapper.java:120)
        at org.mortbay.jetty.Server.handle(Server.java:235)
        at org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java
:374)
        at org.mortbay.jetty.HttpConnection.access$1500(HttpConnection.java
:38)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(
HttpConnection.java:618)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:486)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:196)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:305)
        at org.mortbay.jetty.nio.SelectChannelConnector$HttpEndPoint.run(
SelectChannelConnector.java:733)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(
BoundedThreadPool.java:417)

Help would be extremely appreciated :)
Cheers
 Micke

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message