Return-Path: Delivered-To: apmail-jakarta-commons-httpclient-dev-archive@www.apache.org Received: (qmail 83280 invoked from network); 11 Jul 2004 07:15:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur-2.apache.org with SMTP; 11 Jul 2004 07:15:13 -0000 Received: (qmail 33104 invoked by uid 500); 11 Jul 2004 07:15:12 -0000 Delivered-To: apmail-jakarta-commons-httpclient-dev-archive@jakarta.apache.org Received: (qmail 32928 invoked by uid 500); 11 Jul 2004 07:15:11 -0000 Mailing-List: contact commons-httpclient-dev-help@jakarta.apache.org; run by ezmlm Precedence: bulk List-Unsubscribe: List-Subscribe: List-Help: List-Post: List-Id: "Commons HttpClient Project" Reply-To: "Commons HttpClient Project" Delivered-To: mailing list commons-httpclient-dev@jakarta.apache.org Received: (qmail 32910 invoked by uid 99); 11 Jul 2004 07:15:11 -0000 X-ASF-Spam-Status: No, hits=2.1 required=10.0 tests=DNS_FROM_RFC_ABUSE,FROM_ENDS_IN_NUMS,HTML_40_50,HTML_MESSAGE,HTML_TEXT_AFTER_BODY,HTML_TEXT_AFTER_HTML X-Spam-Check-By: apache.org Received: from [216.136.224.60] (HELO web14310.mail.yahoo.com) (216.136.224.60) by apache.org (qpsmtpd/0.27.1) with SMTP; Sun, 11 Jul 2004 00:15:08 -0700 Message-ID: <20040711071508.93615.qmail@web14310.mail.yahoo.com> Received: from [24.0.200.158] by web14310.mail.yahoo.com via HTTP; Sun, 11 Jul 2004 00:15:08 PDT Date: Sun, 11 Jul 2004 00:15:08 -0700 (PDT) From: Vincent Chain Subject: Cannot use HttpClient to search google To: commons-httpclient-dev@jakarta.apache.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-434424642-1089530108=:93481" X-Virus-Checked: Checked X-Spam-Rating: minotaur-2.apache.org 1.6.2 0/1000/N --0-434424642-1089530108=:93481 Content-Type: text/plain; charset=us-ascii I am encountering an interesting issue, and I guess the issue is probably not within the HttpClient itself, but I haven't figured out how to make it work yet. I am using the 2.0 version. What I try to do is simple enough: I want to use HttpClient to simulate a typical browser request to search in google. For example a query like http://www.google.com/search?hl=en&ie=UTF-8&q=sql+server+trace And I used some code like below client = new HttpClient(); m = new GetMethod("http://www.google.com/search?hl=en&ie=UTF-8&q=sql+server+trace"); s = client.executeMethod(m); Now s is always 403 for me (and this 403 should have nothing to do with Proxy), and the content of the response is basically google saying that the request is forbidden (because it reaches a host that the client is not supposed to)... the response is too large for this email, but it looks like this 403 Forbidden....

Forbidden

Your client does not have permission to get URL /search?hl=en&ie=UTF-8&q=sql+server+trace from this server. (Client IP address: xx.xx.xx.xx)

Please see Google's Terms of Service posted at http://www.google.com/terms_of_service.html .... I guess the main reason is google uses akamai's network to distribute loads. On my server when I do an nslookup of google, I can see that the DNS records returned have very short valid duration: from several seconds to a couple of minutes. I guess this way the browser will be forced to issue another DNS query the next time I do a search. The issue when I use HttpClient however is it always uses a certain IP for www.google.com and seems to ignore the short life of the DNS entry. I think because HttpClient opened a socket to this 'old' IP address google somehow figured it's not a valid request and rejected it. I did a quick check on the HttpClient code and it seems to me the Socket it uses to open the connection is implemented from java.net.Socket (DefaultProtocolSocketFactory::createSocket), so I guess HttpClient is not directly responsible for the problem here... Nevertheless, I wonder if any one having similar issue as I do? Especially considering some of the HttpClient sample codes uses http://www.google.com then it should have similar problems? Thanks a lot for any tips. --------------------------------- Do you Yahoo!? Yahoo! Mail is new and improved - Check it out! --0-434424642-1089530108=:93481--