Mailing-List: contact dev-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpComponents Project" <dev@hc.apache.org>
Date: Sun, 23 Sep 2012 23:41:07 +1100 (NCT)
From: "Oleg Kalnichevski (JIRA)" <jira@apache.org>
To: dev@hc.apache.org
Message-ID: <1170803366.113582.1348404067745.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (HTTPCORE-195) Make it possible to tolerate
 truncated chunk streams
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HTTPCORE-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461407#comment-13461407 ] 

Oleg Kalnichevski commented on HTTPCORE-195:
--------------------------------------------

Ian

Why can't you simply terminate content processing upon catching the exception? Something along these lines?

---
StringBuilder buffer = new StringBuilder();
InputStream instream = entity.getContent();
try {
    Reader reader = new InputStreamReader(instream, charset);
    char[] tmp = new char[1024];
    int l;
    while((l = reader.read(tmp)) != -1) {
        buffer.append(tmp, 0, l);
    }
} catch (MalformedChunkCodingException ignore) {
} finally {
    instream.close();
}
return buffer.toString();
---

Oleg
                
> Make it possible to tolerate truncated chunk streams
> ----------------------------------------------------
>
>                 Key: HTTPCORE-195
>                 URL: https://issues.apache.org/jira/browse/HTTPCORE-195
>             Project: HttpComponents HttpCore
>          Issue Type: Improvement
>          Components: HttpCore NIO
>    Affects Versions: 4.0
>            Reporter: Patrick Moore
>            Priority: Minor
>             Fix For: 4.1-alpha1
>
>         Attachments: chunkValidationDecoupling.patch, HTTPCORE-195.patch
>
>
> Our server is webcrawling.
> We are frequently encountering this issue. We think this might be related to something on the server that we are scanning. But that doesn't matter. We need to handle such cases without exceptions. (From my perspective, such things should generate a debug message -- certainly not an exception that ends processing and throws away the retrieved content! )
> http://stuftpizza.com/ seems to reliably result in this problem
> May be TransferEncoding? http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6
> Either way we need to be able to deal with issues on the other servers.
> {{{
> Date	Mon, 20 Apr 2009 03:56:45 GMT
> Server	Apache/2.2.3 (Red Hat)
> Accept-Ranges	bytes
> Connection	close
> Transfer-Encoding	chunked
> Content-Type	text/html
> '''Request Headers'''
> Host	stuftpizza.com
> User-Agent	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.8) Gecko/2009032608 Firefox/3.0.8
> Accept	text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language	en-us,en;q=0.5
> Accept-Encoding	gzip,deflate
> Accept-Charset	ISO-8859-1,utf-8;q=0.7,*;q=0.7
> Keep-Alive	300
> Connection	keep-alive
> Cookie	
> __utma=47358053.1237981682.1240199754.1240199754.1240199754.1; __utmb=47358053; __utmc=47358053; __utmz
> =47358053.1240199754.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none)
> Cache-Control	max-age=0
> }}}
> {{{
> 20:51:08,768 INFO  [nioEventListener] Request http://stuftpizza.com/ failed with exception.
> org.apache.http.MalformedChunkCodingException: Truncated chunk
> 	at org.apache.http.impl.nio.codecs.ChunkDecoder.read(ChunkDecoder.java:203)
> 	at org.apache.http.nio.util.SimpleInputBuffer.consumeContent(SimpleInputBuffer.java:60)
> 	at org.apache.http.nio.entity.BufferingNHttpEntity.consumeContent(BufferingNHttpEntity.java:72)
> 	at org.apache.http.nio.protocol.AsyncNHttpClientHandler.inputReady(AsyncNHttpClientHandler.java:236)
> 	at org.apache.http.nio.protocol.BufferingHttpClientHandler.inputReady(BufferingHttpClientHandler.java:118)
> 	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:178)
> 	at org.apache.http.impl.nio.DefaultClientIOEventDispatch.inputReady(DefaultClientIOEventDispatch.java:146)
> 	at com.amplafi.iomanagement.http.UniversalIOEventDispatch.inputReady(UniversalIOEventDispatch.java:133)
> 	at $IOEventDispatch_120c19cd1c7.inputReady($IOEventDispatch_120c19cd1c7.java)
> 	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:153)
> 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:314)
> 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:294)
> 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
> 	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:96)
> 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:556)
> 	at java.lang.Thread.run(Thread.java:637)
> }}}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org