hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: Incredibly slow gzip downloads
Date Sat, 15 Feb 2014 16:13:15 GMT
On Sat, 2014-02-15 at 17:06 +0100, Sebastiano Vigna wrote:
> On 15 Feb 2014, at 12:47 PM, Oleg Kalnichevski <olegk@apache.org> wrote:
> > The problem mostly likely has been introduced by HTTPCLIENT-1432 [1]. I
> > reviewed the patch once more and could not find anything obviously wrong
> > with it. Try reverting the changes introduced by HTTPCLIENT-1432 and see
> Subclassing InputStream without overriding the multi-byte read() method is a recipe for
disaster... the inherited method does a byte-by-byte read. You can see what's happening in
this hprof trace:
>         java.util.zip.Inflater.inflateBytes(Inflater.java:Unknown line)
>         java.util.zip.Inflater.inflate(Inflater.java:259)
>         java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
>         java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116)
>         java.util.zip.InflaterInputStream.read(InflaterInputStream.java:122)
>         org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:56)
>         java.io.InputStream.read(InputStream.java:179)
>         it.unimi.di.law.warc.util.InspectableCachedHttpEntity.copyContent(InspectableCachedHttpEntity.java:67)
> copyContent() would love to read(byte[],int,int) in a buffer, but since LazyDecompressingInputStream
doesn't override it it invokes instead the read-byte-by-byte inherited method in InputStream,
which in turn now calls for each byte the one-byte read() method from LazyDecompressingInputStream,
which invokes the one-byte read method from InflaterInputStream, which does a multi-byte,
length-one read from GZIPInputStream, which unleashes a similar call on InflaterInputStream,
which unfortunately makes a similar read using the native inflateBytes() method. 
> The result is a 10-50x decrease in speed, but I think that a trivial override of read(byte[],int,int)
in LazyDecompressingInputStream will solve the problem.


Could you please raise a JIRA for this defect and tag it as regression?


To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org

View raw message