hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sam Berlin <sber...@limepeer.com>
Subject Re: Does HttpClient decompress compressed HTTP transfers?
Date Mon, 17 Nov 2003 18:24:21 GMT
It'd be rather easy to wrap the streams in a DeflaterOutputStream or an 
InflaterInputStream.  Of course, due to limitations in Java's deflate 
compression, one must extend DeflaterOutputStream to allow true stream 
deflation.  The problem with the current implementation is that there is 
no way to flush a partially deflated stream -- deflate waits until it 
reaches an optimal spot to actually perform the deflation and do the 
flush.  You can see  
http://developer.java.sun.com/developer/bugParade/bugs/4255743.html and 
http://developer.java.sun.com/developer/bugParade/bugs/4206909.html .  
In LimeWire we worked around this by using a workaround listed on those 
pages: having a class 'CompressingOutputStream' that extends 
DeflaterOutputStream, using the following as the flush method:

    private static final byte [] EMPTYBYTEARRAY = new byte [0];

    /**
     * Ensure all remaining data will be output.
     */
    public void flush() throws IOException {
        if( def.finished() ) return;
       
        /**
         * Now this is tricky: We force the Deflater to flush
         * its data by switching compression level.
         * As yet, a perplexingly simple workaround for
         * 
http://developer.java.sun.com/developer/bugParade/bugs/4255743.html
         */
        def.setInput(EMPTYBYTEARRAY, 0, 0);

        def.setLevel(Deflater.NO_COMPRESSION);
        deflate();

        def.setLevel(Deflater.DEFAULT_COMPRESSION);
        deflate();

        super.flush();
    }


The other thing you have to be careful of when using 
DeflaterOutputStream or InflaterInputStream is spurious 
NullPointerExceptions from native code if the connection happens to 
close while a deflate/inflate is being performed.  We worked around that 
particular problem by extending the 'read' of the InflaterInputStream 
and the 'deflate' call of the DeflaterOutputStream to catch an NPE and 
rethrow an IOX.

Thanks,
 Sam

otisg wrote:

>I'm not too familiar with HttpClient source code any more, but
>it seems to me that this should be easy to add.  Please correct
>me if I am wrong.
>I will be using HttpClient to pull large amounts of data from
>the web in a few months, and allowing compressed content may
>translate to less bandwidth and a lower hosting bill :)
>
>Otis
>
>
>
>________________________________________________
>Get your own "800" number
>Voicemail, fax, email, and a lot more
>http://www.ureach.com/reg/tag
>
>
>---- On Mon, 17 Nov 2003, =?ISO-8859-15?Q?Sven_K=F6hler?=
>(skoehler@upb.de) wrote:
>
>  
>
>>Hi,
>>
>>If i send the right accept-encoding headers, the web-server
>>    
>>
>may answer 
>  
>
>>with a gzip or deflate compressed stream. Does HttpClient
>>    
>>
>decompress it? 
>  
>
>>If yes, how can i turn that off?
>>
>>
>>
>>    
>>
>---------------------------------------------------------------------
>  
>
>>To unsubscribe, e-mail:
>>    
>>
>commons-httpclient-dev-unsubscribe@jakarta.apache.org
>  
>
>>For additional commands, e-mail:
>>    
>>
>commons-httpclient-dev-help@jakarta.apache.org
>  
>
>>
>>    
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org
>
>
>.
>
>  
>



---------------------------------------------------------------------
To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org


Mime
View raw message