hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Spencer Lee" <spe...@gmail.com>
Subject Unable to retreive all data of a large (>30 MB) file.
Date Tue, 01 Aug 2006 17:05:50 GMT
Hello,

Just wondering if anyone has had problems retreiving data all the data of a
large file.

Here's basically the code I've been running...

...
            HttpClient httpclient = new HttpClient(params);
            httpclient.getHostConfiguration().setHost("myhost", 80, "http");
            httpclient.getParams().setCookiePolicy(
CookiePolicy.BROWSER_COMPATIBILITY);
            httpclient.getHostConfiguration ().setProxy("webproxy", 80);
...
            // Create a method instance.
            GetMethod method = new GetMethod(myurl);
            DefaultHttpMethodRetryHandler retryhandler = new
DefaultHttpMethodRetryHandler(10, false);
            HttpMethodParams method_params = method.getParams();
            method_params.setParameter(HttpMethodParams.RETRY_HANDLER,
retryhandler);
...
            // executing the method
            int statusCode = httpclient.executeMethod(method);

            if(statusCode != HttpStatus.SC_OK) {
                System.out.println("ERROR: Status code " + statusCode);
                return;
            }

            // Read the response body.
            InputStream inpStream = method.getResponseBodyAsStream();

            BufferedInputStream bufinstrm = new
BufferedInputStream(inpStream);
            long totalBytesRead = 0;
            while(true) {
                byte[] buffer = new byte[12*1024];
                int bytesRead;

                bytesRead = bufinstrm.read(buffer);
                if(bytesRead == -1) {
                    System.err.println("Read -1.  Read complete.  Total
Bytes Read: " + totalBytesRead + " bytes");
                    break;
                }

    //            bufotstrm.write(buffer, 0, bytesRead);
                totalBytesRead += bytesRead;
                System.err.println("Total Bytes Read: " + totalBytesRead + "
bytes");
            }

            bufinstrm.close();

            // close
            method.releaseConnection();



The byte size of the file is 36,197,082.  But I am only able to retreival
all 36,197,082 bytes once in awhile.  On most occasions my
BufferedInputStream (bufinstrm) will return a -1 to signal EOF before all
bytes are read.

eg.
  ...
  Total Bytes Read: 26796032 bytes
  Read -1.  Read complete.  Total Bytes Read: 26796032 bytes
  >

Anyone come across anything similar to this before?

Thanks!

Spencer

Also...
* I'm using release commons-httpclient-3.0.1
* I am not seeing any abnormal messages printed to the logs.
* I am working through a web proxy server.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message