hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thibaut (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HTTPCLIENT-1257) Header location automatically converted to ASCII even though location can contain UTF-8 encoded urls
Date Mon, 12 Nov 2012 15:51:12 GMT

    [ https://issues.apache.org/jira/browse/HTTPCLIENT-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495352#comment-13495352
] 

Thibaut commented on HTTPCLIENT-1257:
-------------------------------------

If I force httpclient to use a utf-8 encoding, I get another exception on the following url:
http://forum.tour-magazin.de/showthread.php?t=226439

The redirect works fine with US-ASCII parsing.


2012-11-12 16:46:01,216 DEBUG http.wire: << "HTTP/1.1 301 Moved Permanently[\r][\n]"
[main]
2012-11-12 16:46:01,217 DEBUG http.wire: << "Date: Mon, 12 Nov 2012 15:45:51 GMT[\r][\n]"
[main]
2012-11-12 16:46:01,217 DEBUG http.wire: << "Server: Apache/2.2.12 (Ubuntu)[\r][\n]"
[main]
2012-11-12 16:46:01,217 DEBUG http.wire: << "X-Powered-By: PHP/5.2.10-2ubuntu6.10[\r][\n]"
[main]
2012-11-12 16:46:01,217 DEBUG http.wire: << "Cache-Control: no-cache, must-revalidate[\r][\n]"
[main]
2012-11-12 16:46:01,217 DEBUG http.wire: << "Expires: 0[\r][\n]" [main]
2012-11-12 16:46:01,218 DEBUG http.wire: << "Set-Cookie: bb_lastvisit=1352735151; expires=Tue,
12-Nov-2013 15:45:51 GMT; path=/[\r][\n]" [main]
2012-11-12 16:46:01,218 DEBUG http.wire: << "Set-Cookie: bb_lastactivity=0; expires=Tue,
12-Nov-2013 15:45:51 GMT; path=/[\r][\n]" [main]
2012-11-12 16:46:01,218 DEBUG http.wire: << "Cache-Control: private, post-check=0, pre-check=0,
max-age=0[\r][\n]" [main]
2012-11-12 16:46:01,218 DEBUG http.wire: << "Pragma: no-cache[\r][\n]" [main]
2012-11-12 16:46:01,219 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:50130<->212.8.200.231:80
closed [main]
2012-11-12 16:46:01,219 DEBUG client.DefaultHttpClient: Closing the connection. [main]
2012-11-12 16:46:01,219 DEBUG conn.DefaultClientConnection: Connection 0.0.0.0:50130<->212.8.200.231:80
closed [main]
2012-11-12 16:46:01,219 INFO  client.DefaultHttpClient: I/O exception (java.nio.charset.MalformedInputException)
caught when processing request: Input length = 1 [main]
2012-11-12 16:46:01,219 DEBUG client.DefaultHttpClient: Input length = 1 [main]
java.nio.charset.MalformedInputException: Input length = 1
	at java.nio.charset.CoderResult.throwException(Unknown Source)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.handleDecodingResult(AbstractSessionInputBuffer.java:384)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.appendDecoded(AbstractSessionInputBuffer.java:371)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.lineFromReadBuffer(AbstractSessionInputBuffer.java:349)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:268)
	at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
	at org.apache.http.impl.io.AbstractMessageParser.parseHeaders(AbstractMessageParser.java:186)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
	at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:712)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
	at com.trendiction.modules.fetch.WebFetch.fetchUrlInternal2(WebFetch.java:821)
	at com.trendiction.modules.fetch.WebFetch.fetchUrlInternal(WebFetch.java:605)
	at com.trendiction.modules.fetch.WebFetch.fetchUrlWithCRTest(WebFetch.java:502)
	at com.trendiction.modules.fetch.WebFetch.fetchUrl(WebFetch.java:474)
	at com.trendiction.modules.fetch.Fetch.fetchWithoutRedirect(Fetch.java:647)
	at com.trendiction.modules.fetch.Fetch.fetch(Fetch.java:239)
	at com.trendiction.modules.fetch.Fetch.fetch(Fetch.java:172)
	at com.trendiction.modules.fetch.Fetch.fetch(Fetch.java:149)
	at com.trendiction.modules.fetch.Fetch.fetch(Fetch.java:143)
	at com.trendiction.modules.fetch.Fetch.fetch(Fetch.java:137)
	at com.trendiction.modules.html.TestHtmlFetching.testTourMagazinUrlFilter(TestHtmlFetching.java:345)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)



So what do you suggest? use two different instances of httpclient and when one fails, try
the same request with the other one?

Again I'm for having the headers saved as bytearray and being able to specify the encoding
when accessing the header fields.





                
> Header location automatically converted to ASCII even though location can contain UTF-8
encoded urls
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HTTPCLIENT-1257
>                 URL: https://issues.apache.org/jira/browse/HTTPCLIENT-1257
>             Project: HttpComponents HttpClient
>          Issue Type: Bug
>          Components: HttpClient
>    Affects Versions: 4.2.2
>            Reporter: Thibaut
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I'm trying to fetch:
> http://handheld.vn/content.php?4052-Đánh-giá-máy-tính-bảng-Kindle-Fire-HD-7-inch
> Which returns:
> 2012-10-29 18:54:29,355 DEBUG http.wire: << "HTTP/1.1 303 See Other[\r][\n]" [main]
> 2012-10-29 18:54:29,355 DEBUG http.wire: << "Date: Mon, 29 Oct 2012 17:55:57 GMT[\r][\n]"
[main]
> 2012-10-29 18:54:29,355 DEBUG http.wire: << "Server: Apache[\r][\n]" [main]
> 2012-10-29 18:54:29,355 DEBUG http.wire: << "Expires: Thu, 19 Nov 1981 08:52:00
GMT[\r][\n]" [main]
> 2012-10-29 18:54:29,356 DEBUG http.wire: << "Cache-Control: no-store, no-cache,
must-revalidate, post-check=0, pre-check=0[\r][\n]" [main]
> 2012-10-29 18:54:29,356 DEBUG http.wire: << "Pragma: no-cache[\r][\n]" [main]
> 2012-10-29 18:54:29,356 DEBUG http.wire: << "Set-Cookie: bb_lastactivity=0; expires=Tue,
29-Oct-2013 17:55:57 GMT; path=/[\r][\n]" [main]
> 2012-10-29 18:54:29,356 DEBUG http.wire: << "Location: http://handheld.vn/content/4052-????nh-gi??-m??y-t??nh-b???ng-Kindle-Fire-HD-7-inch[\r][\n]"
[main]
> 2012-10-29 18:54:29,357 DEBUG http.wire: << "Content-Length: 0[\r][\n]" [main]
> 2012-10-29 18:54:29,357 DEBUG http.wire: << "Connection: close[\r][\n]" [main]
> 2012-10-29 18:54:29,357 DEBUG http.wire: << "Content-Type: text/html[\r][\n]" [main]
> 2012-10-29 18:54:29,357 DEBUG http.wire: << "[\r][\n]" [main]
> 2012-10-29 18:54:29,357 DEBUG conn.DefaultClientConnection: Receiving response: HTTP/1.1
303 See Other [main]
> 2012-10-29 18:54:29,357 DEBUG http.headers: << HTTP/1.1 303 See Other [main]
> 2012-10-29 18:54:29,358 DEBUG http.headers: << Date: Mon, 29 Oct 2012 17:55:57
GMT [main]
> 2012-10-29 18:54:29,358 DEBUG http.headers: << Server: Apache [main]
> 2012-10-29 18:54:29,358 DEBUG http.headers: << Expires: Thu, 19 Nov 1981 08:52:00
GMT [main]
> 2012-10-29 18:54:29,358 DEBUG http.headers: << Cache-Control: no-store, no-cache,
must-revalidate, post-check=0, pre-check=0 [main]
> 2012-10-29 18:54:29,358 DEBUG http.headers: << Pragma: no-cache [main]
> 2012-10-29 18:54:29,358 DEBUG http.headers: << Set-Cookie: bb_lastactivity=0; expires=Tue,
29-Oct-2013 17:55:57 GMT; path=/ [main]
> 2012-10-29 18:54:29,358 DEBUG http.headers: << Location: http://handheld.vn/content/4052-Đánh-giá-máy-tính-bảng-Kindle-Fire-HD-7-inch
[main]
> 2012-10-29 18:54:29,358 DEBUG http.headers: << Content-Length: 0 [main]
> 2012-10-29 18:54:29,358 DEBUG http.headers: << Connection: close [main]
> 2012-10-29 18:54:29,359 DEBUG http.headers: << Content-Type: text/html [main]
> Unfortunately I can't get the resolve Url through the following code:
> Header locationHeader = response.getFirstHeader("location");
> which will return http://handheld.vn/content/4052-Đánh-giá-máy-tính-bảng-Kindle-Fire-HD-7-inch
> The header has already been extracted in the wrong content encoding. I will never be
able to get the redirect url!
> I understand that this is not RFC normalised behavior, but the above url and redirect
works fine in all browsers.
> Is it possible to access the raw header (byte array) so that I can chose the encoding
on my own? This would help a lot. Or a parameter to optionally specify the encoding when fetching
a header value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message