commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Renouf <eren...@skaion.com>
Subject Re: httpclient local address trouble
Date Tue, 07 Dec 2004 18:21:48 GMT
Guess it didn't like my attachment, so here's some of the wire log:

2004/12/07 12:16:11:115 EST [DEBUG] DefaultHttpParams - Set parameter
http.useragent = Jakarta Commons-HttpClient/3.0-beta1
2004/12/07 12:16:11:120 EST [DEBUG] DefaultHttpParams - Set parameter
http.protocol.version = HTTP/1.1
2004/12/07 12:16:11:126 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection-manager.class = class
org.apache.commons.httpclient.SimpleHttpConnectionManager
2004/12/07 12:16:11:127 EST [DEBUG] DefaultHttpParams - Set parameter
http.protocol.cookie-policy = rfc2109
2004/12/07 12:16:11:128 EST [DEBUG] DefaultHttpParams - Set parameter
http.protocol.element-charset = US-ASCII
2004/12/07 12:16:11:128 EST [DEBUG] DefaultHttpParams - Set parameter
http.protocol.content-charset = ISO-8859-1
2004/12/07 12:16:11:132 EST [DEBUG] DefaultHttpParams - Set parameter
http.method.retry-handler =
org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@863399
2004/12/07 12:16:11:134 EST [DEBUG] DefaultHttpParams - Set parameter
http.dateparser.patterns = [EEE, dd MMM yyyy HH:mm:ss zzz, EEEE, dd-MMM-yy
HH:mm:ss zzz, EEE MMM d HH:mm:ss yyyy, EEE, dd-MMM-yyyy HH:mm:ss z, EEE,
dd-MMM-yyyy HH-mm-ss z, EEE, dd MMM yy HH:mm:ss z, EEE dd-MMM-yyyy HH:mm:ss z,
EEE dd MMM yyyy HH:mm:ss z, EEE dd-MMM-yyyy HH-mm-ss z, EEE dd-MMM-yy HH:mm:ss
z, EEE dd MMM yy HH:mm:ss z, EEE,dd-MMM-yy HH:mm:ss z, EEE,dd-MMM-yyyy HH:mm:ss
z, EEE, dd-MM-yyyy HH:mm:ss z]
2004/12/07 12:16:11:324 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection-manager.max-per-host = {HostConfiguration[]=100}
2004/12/07 12:16:11:326 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection-manager.max-total = 100
2004/12/07 12:16:11:326 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection.timeout = 5000
2004/12/07 12:16:11:327 EST [DEBUG] IdleConnectionHandler - Checking for
connections, idleTimeout: 1102439766327
2004/12/07 12:16:11:333 EST [DEBUG] HttpClient - Java version: 1.5.0
2004/12/07 12:16:11:334 EST [DEBUG] HttpClient - Java vendor: Sun Microsystems Inc.
2004/12/07 12:16:11:334 EST [DEBUG] HttpClient - Java class path:
Bots.jar:commons-httpclient-3.0-beta1.jar:commons-logging-api.jar:commons-codec-1.3.jar
2004/12/07 12:16:11:334 EST [DEBUG] HttpClient - Operating system name: Linux
2004/12/07 12:16:11:335 EST [DEBUG] HttpClient - Operating system architecture: i386
2004/12/07 12:16:11:335 EST [DEBUG] HttpClient - Operating system version:
2.4.20-8-ipswap
2004/12/07 12:16:11:922 EST [DEBUG] HttpClient - SUN 1.5: SUN (DSA key/parameter
generation; DSA signing; SHA-1, MD5 digests; SecureRandom; X.509 certificates;
JKS keystore; PKIX CertPathValidator; PKIX CertPathBuilder; LDAP, Collection
CertStores)
2004/12/07 12:16:11:923 EST [DEBUG] HttpClient - SunRsaSign 1.5: Sun RSA
signature provider
2004/12/07 12:16:11:923 EST [DEBUG] HttpClient - SunJSSE 1.5: Sun JSSE
provider(PKCS12, SunX509 key/trust factories, SSLv3, TLSv1)
2004/12/07 12:16:11:923 EST [DEBUG] HttpClient - SunJCE 1.5: SunJCE Provider
(implements RSA, DES, Triple DES, AES, Blowfish, ARCFOUR, RC2, PBE,
Diffie-Hellman, HMAC)
2004/12/07 12:16:11:924 EST [DEBUG] HttpClient - SunJGSS 1.0: Sun (Kerberos v5)
2004/12/07 12:16:11:924 EST [DEBUG] HttpClient - SunSASL 1.5: Sun SASL
provider(implements client mechanisms for: DIGEST-MD5, GSSAPI, EXTERNAL, PLAIN,
CRAM-MD5; server mechanisms for: DIGEST-MD5, GSSAPI, CRAM-MD5)
2004/12/07 12:16:11:977 EST [DEBUG] MultiThreadedHttpConnectionManager -
HttpConnectionManager.getConnection:  config =
HostConfiguration[host=http://www.cia.gov, localAddress=/100.20.140.113,
params=org.apache.commons.httpclient.params.HostParams@136228], timeout = 0
2004/12/07 12:16:11:981 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection-manager.max-per-host = {HostConfiguration[]=100}
2004/12/07 12:16:11:982 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection-manager.max-total = 100
2004/12/07 12:16:11:982 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection.timeout = 5000
2004/12/07 12:16:11:982 EST [DEBUG] IdleConnectionHandler - Checking for
connections, idleTimeout: 1102439766982
2004/12/07 12:16:11:984 EST [DEBUG] MultiThreadedHttpConnectionManager -
HttpConnectionManager.getConnection:  config =
HostConfiguration[host=http://www.thevillage.int, localAddress=/100.20.140.114,
params=org.apache.commons.httpclient.params.HostParams@19bd03e], timeout = 0
2004/12/07 12:16:11:986 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection-manager.max-per-host = {HostConfiguration[]=100}
2004/12/07 12:16:11:987 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection-manager.max-total = 100
2004/12/07 12:16:11:987 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection.timeout = 5000
2004/12/07 12:16:11:987 EST [DEBUG] IdleConnectionHandler - Checking for
connections, idleTimeout: 1102439766987
2004/12/07 12:16:11:991 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection-manager.max-per-host = {HostConfiguration[]=100}
2004/12/07 12:16:11:991 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection-manager.max-total = 100
2004/12/07 12:16:11:992 EST [DEBUG] DefaultHttpParams - Set parameter
http.connection.timeout = 5000
2004/12/07 12:16:11:992 EST [DEBUG] IdleConnectionHandler - Checking for
connections, idleTimeout: 1102439766992
2004/12/07 12:16:11:995 EST [DEBUG] MultiThreadedHttpConnectionManager -
HttpConnectionManager.getConnection:  config =
HostConfiguration[host=http://www.mi5-5.gov.uk, localAddress=/100.20.140.112,
params=org.apache.commons.httpclient.params.HostParams@bfbdb0], timeout = 0
2004/12/07 12:16:11:997 EST [DEBUG] MultiThreadedHttpConnectionManager -
Allocating new connection,
hostConfig=HostConfiguration[host=http://www.mi5-5.gov.uk,
localAddress=/100.20.140.112,
params=org.apache.commons.httpclient.params.HostParams@3e86d0]
2004/12/07 12:16:12:000 EST [DEBUG] MultiThreadedHttpConnectionManager -
Allocating new connection, hostConfig=HostConfiguration[host=http://www.cia.gov,
localAddress=/100.20.140.113,
params=org.apache.commons.httpclient.params.HostParams@19fcc69]
2004/12/07 12:16:12:000 EST [DEBUG] MultiThreadedHttpConnectionManager -
HttpConnectionManager.getConnection:  config =
HostConfiguration[host=http://www.openpost.com, localAddress=/100.20.140.111,
params=org.apache.commons.httpclient.params.HostParams@253498], timeout = 0
2004/12/07 12:16:12:001 EST [DEBUG] MultiThreadedHttpConnectionManager -
Allocating new connection,
hostConfig=HostConfiguration[host=http://www.openpost.com,
localAddress=/100.20.140.111,
params=org.apache.commons.httpclient.params.HostParams@9fef6f]
2004/12/07 12:16:12:001 EST [DEBUG] MultiThreadedHttpConnectionManager -
Allocating new connection,
hostConfig=HostConfiguration[host=http://www.thevillage.int,
localAddress=/100.20.140.114,
params=org.apache.commons.httpclient.params.HostParams@209f4e]
2004/12/07 12:16:12:218 EST [DEBUG] HttpConnection - Open connection to
www.openpost.com:80
2004/12/07 12:16:12:234 EST [DEBUG] HttpConnection - Open connection to
www.thevillage.int:80
2004/12/07 12:16:12:238 EST [DEBUG] HttpConnection - Open connection to
www.cia.gov:80
2004/12/07 12:16:12:248 EST [DEBUG] header - >> "GET / HTTP/1.1[\r][\n]"
2004/12/07 12:16:12:250 EST [DEBUG] HttpMethodBase - Adding Host request header
2004/12/07 12:16:12:218 EST [DEBUG] HttpConnection - Open connection to
www.mi5-5.gov.uk:80
2004/12/07 12:16:12:270 EST [DEBUG] header - >> "GET / HTTP/1.1[\r][\n]"
2004/12/07 12:16:12:270 EST [DEBUG] HttpMethodBase - Adding Host request header
2004/12/07 12:16:12:272 EST [DEBUG] header - >> "GET / HTTP/1.1[\r][\n]"
2004/12/07 12:16:12:273 EST [DEBUG] HttpMethodBase - Adding Host request header
2004/12/07 12:16:12:272 EST [DEBUG] header - >> "GET / HTTP/1.1[\r][\n]"
2004/12/07 12:16:12:275 EST [DEBUG] HttpMethodBase - Adding Host request header
2004/12/07 12:16:12:284 EST [DEBUG] header - >> "User-Agent: Jakarta
Commons-HttpClient/3.0-beta1[\r][\n]"
2004/12/07 12:16:12:288 EST [DEBUG] header - >> "User-Agent: Jakarta
Commons-HttpClient/3.0-beta1[\r][\n]"
2004/12/07 12:16:12:288 EST [DEBUG] header - >> "Host: www.thevillage.int[\r][\n]"
2004/12/07 12:16:12:309 EST [DEBUG] header - << "Server: Apache/1.3.23 (Unix) 
(Red-Hat/Linux) mod_ssl/2.8.7 OpenSSL/0.9.6b DAV/1.0.3 PHP/4.1.2
mod_perl/1.26[\r][\n]"
2004/12/07 12:16:12:309 EST [DEBUG] header - << "Last-Modified: Tue, 11 May 2004
18:46:58 GMT[\r][\n]"
2004/12/07 12:16:12:309 EST [DEBUG] header - << "ETag: "1525dc-6f-40a11fa2"[\r][\n]"
2004/12/07 12:16:12:310 EST [DEBUG] header - << "Accept-Ranges: bytes[\r][\n]"
2004/12/07 12:16:12:311 EST [DEBUG] header - << "Content-Length: 111[\r][\n]"
2004/12/07 12:16:12:311 EST [DEBUG] header - << "Connection: close[\r][\n]"
2004/12/07 12:16:12:312 EST [DEBUG] header - << "Content-Type: text/html[\r][\n]"
2004/12/07 12:16:12:320 EST [DEBUG] header - << "HTTP/1.1 200 OK[\r][\n]"
2004/12/07 12:16:12:321 EST [DEBUG] header - << "Date: Tue, 07 Dec 2004 17:16:12
GMT[\r][\n]"
2004/12/07 12:16:12:322 EST [DEBUG] header - << "Server: Apache/1.3.23 (Unix) 
(Red-Hat/Linux) mod_ssl/2.8.7 OpenSSL/0.9.6b DAV/1.0.3 PHP/4.1.2
mod_perl/1.26[\r][\n]"
2004/12/07 12:16:12:322 EST [DEBUG] header - << "Connection: close[\r][\n]"
2004/12/07 12:16:12:323 EST [DEBUG] header - << "Transfer-Encoding: chunked[\r][\n]"
2004/12/07 12:16:12:324 EST [DEBUG] header - << "HTTP/1.1 200 OK[\r][\n]"
2004/12/07 12:16:12:326 EST [DEBUG] header - << "Date: Tue, 07 Dec 2004 17:16:12
GMT[\r][\n]"
2004/12/07 12:16:12:326 EST [DEBUG] header - << "Server: Apache/1.3.23 (Unix) 
(Red-Hat/Linux) mod_ssl/2.8.7 OpenSSL/0.9.6b DAV/1.0.3 PHP/4.1.2
mod_perl/1.26[\r][\n]"
2004/12/07 12:16:12:327 EST [DEBUG] header - << "Last-Modified: Mon, 15 Nov 2004
19:09:46 GMT[\r][\n]"
2004/12/07 12:16:12:327 EST [DEBUG] header - << "ETag:
"c6f42-4942-4198fefa"[\r][\n]"
2004/12/07 12:16:12:328 EST [DEBUG] header - << "Accept-Ranges: bytes[\r][\n]"
2004/12/07 12:16:12:328 EST [DEBUG] header - << "Content-Length: 18754[\r][\n]"
2004/12/07 12:16:12:329 EST [DEBUG] header - << "Connection: close[\r][\n]"
2004/12/07 12:16:12:329 EST [DEBUG] header - << "Content-Type: text/html[\r][\n]"
2004/12/07 12:16:12:331 EST [DEBUG] header - << "HTTP/1.1 200 OK[\r][\n]"
2004/12/07 12:16:12:323 EST [DEBUG] header - << "Content-Type: text/html[\r][\n]"
2004/12/07 12:16:12:339 EST [DEBUG] header - << "Date: Tue, 07 Dec 2004 17:16:12
GMT[\r][\n]"
2

Quoting Eric Renouf <erenouf@skaion.com>:

> Ok, thanks for taking the time to look.  Here's a few pieces of my code:
> 
> This is from the constructor for the class:
>         hostConf = new HostConfiguration();
>         hostConf.setHost(cur);
>         if(ip != null) {
>             try {
>                 hostConf.setLocalAddress(InetAddress.getByName(ip));
>             } catch(Exception e) {
>                 System.err.println("Couldn't get that address: " +
> e.getMessage());
>             }
>         }
>         MultiThreadedHttpConnectionManager manager = new
> MultiThreadedHttpConnectionManager();
>         HttpConnectionManagerParams p = manager.getParams();
>         p.setDefaultMaxConnectionsPerHost(100); // TODO Should be
> configurable
> from file
>         p.setMaxTotalConnections(100); // TOOD Should be configurable from
> file
>         p.setConnectionTimeout(5000); // TODO should be configurable from
> file
>         manager.setParams(p);
>         manager.closeIdleConnections(5000); // TODO should be configurable
> from file
>         browser = new HttpClient(manager);
>         browser.setHostConfiguration(hostConf);
> 
> 
> And this is where I try to actually do the request:
> 
>         GetMethod m = new GetMethod();
>         try {
>             m.setPath(cur.getPath());
>             try {
>                 browser.executeMethod(hostConf, m);
>                 if(m.getStatusCode() != 200) {
>                     warn("Got Status code " +
> String.valueOf(m.getStatusCode())
> + " for " + cur);
>                     throw new NoPageException();
>                 }
>                 host = hostConf.getHostURL();
>                 links = getLinks(m);
>             } catch(HttpException hEx) {
>                 System.out.println("Http Exception: " + hEx.getMessage());
>             } catch(IOException ioEx) {
>                 System.out.println("IOException while browsing: " +
> ioEx.getMessage());
>             } finally {
>                 m.releaseConnection();
>             }
> 
> I'm attaching the wire log, or at least the first few hundred lines from it. 
> If
> I trimmed something you need to see let me know.  Thanks for your time and
> help.
> 
> Eric
> 
> Quoting Michael Becke <becke@u.washington.edu>:
> 
> > Hi Eric,
> > 
> > I can't think of anything at the moment that would cause this.  If you 
> > post a sample of the code you are using and a wire log showing the 
> > problem we should be able to track down the problem.
> > 
> > <http://jakarta.apache.org/commons/httpclient/3.0/logging.html>
> > 
> > Mike
> > 
> > On Dec 6, 2004, at 7:43 PM, Eric Renouf wrote:
> > 
> > > Hi,
> > >
> > > I'm trying to setup an application that will run a bunch of 
> > > HttpClient's from
> > > the same machine, simulating a bunch of different "real" users.  One 
> > > thing that
> > > would make the more real is to have them coming from different IP 
> > > addresses,
> > > which are all aliased on the machine already.  I tried creating an
> > > HostConfiguration object and calling
> > > hostConf.setLocalAddress(InetAddress.getByName(ip)); for each IP.  
> > > Sometimes
> > > this seems to work, but sometimes not.  When I watch the Apache logs 
> > > I'll
> > > sometimes see the IPs that I'm trying to use but more often than not I 
> > > see the
> > > default address for the host.  The line above does not raise an 
> > > exception, and
> > > if I call hostConf.getLocalAddress later it gives me the IP back (with 
> > > a / in
> > > front of it).
> > >
> > > I try both calling the setHostConfiguration on the HttpClient object 
> > > and also
> > > passing it in when I call executeMethod (GetMethod only being used).  
> > > I'm using
> > > the commons-httpclient-3.0-beta1 package.
> > >
> > > Can someone help point me in the direction for what I can do to figure 
> > > out what
> > > I'm doing wrong here?  I've tried googling "httpclient local address" 
> > > and tried
> > > similar searches through the mailing list archives but haven't found 
> > > anything
> > > too promising yet.  Any help would be very much appreciated.
> > >
> > > Eric
> > > ___________________________________________________________
> > > This mail sent using ToadMail -- Web based e-mail @ ToadNet
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail: commons-user-help@jakarta.apache.org
> > >
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: commons-user-help@jakarta.apache.org
> > 
> 
> 
> 
> ___________________________________________________________
> This mail sent using ToadMail -- Web based e-mail @ ToadNet
> 
> 



___________________________________________________________
This mail sent using ToadMail -- Web based e-mail @ ToadNet

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org


Mime
View raw message