Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Message-ID: <938930242.1261475549419.JavaMail.jira@brutus>
Date: Tue, 22 Dec 2009 09:52:29 +0000 (UTC)
From: "Filipe Manana (JIRA)" <jira@apache.org>
To: dev@couchdb.apache.org
Subject: [jira] Commented: (COUCHDB-583) adding ?compression=(gzip|deflate)
 optional parameter to the attachment download API
In-Reply-To: <844257352.1259509400633.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/COUCHDB-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793569#action_12793569 ] 

Filipe Manana commented on COUCHDB-583:
---------------------------------------

Hi Adam,

I see. I am just thinking about how to do it without causing backward incompatibility.

Currently couch_file:append_term[_md5]/2 calls term_to_binary and precedes it with a 32 bits header and an optional md5 digest. Then the 32bits header + optional term md5 + term_to_binary(Term) is appended to the end of the DB file. 

The high order bit of this header indicates whether an md5 hash follows the header (and preceding the serialized term).
Without looking deeply into the code, I think about adding an extra bit to the header which indicates if the term is compressed or not.

Of course this implies adding a new DB header version value, etc. Not so straightforward as attachment compression.

An (ugly) alternative I see is not adding a new header bit and when reading a serialized term from the DB file, always gunzip it and catch an exception:

3> catch(zlib:gunzip(<<"hello world">>)).
{'EXIT',{data_error,[{zlib,call,3},
                     {zlib,inflate,2},
                     {zlib,gunzip,1},
                     {erl_eval,do_apply,5},
                     {erl_eval,expr,5},
                     {shell,exprs,6},
                     {shell,eval_exprs,6},
                     {shell,eval_loop,3}]}}

The issue is that the data_error exception might not mean that the data is not gzip compressed.

If using an extra header bit, than why not add a few more bits that will be reserved for future features. A little bit like most protocol RFCs do, they reserve a few bits in an header for future usage :)

What's your opinion?

cheers


> adding ?compression=(gzip|deflate) optional parameter to the attachment download API
> ------------------------------------------------------------------------------------
>
>                 Key: COUCHDB-583
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-583
>             Project: CouchDB
>          Issue Type: New Feature
>          Components: HTTP Interface
>         Environment: CouchDB trunk revision 885240
>            Reporter: Filipe Manana
>         Attachments: couchdb-583-trunk-3rd-try.patch, couchdb-583-trunk-4th-try-trunk.patch, couchdb-583-trunk-5th-try.patch, couchdb-583-trunk-6th-try.patch, jira-couchdb-583-1st-try-trunk.patch, jira-couchdb-583-2nd-try-trunk.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The following new feature is added in the patch following this ticket creation.
> A new optional http query parameter "compression" is added to the attachments API.
> This parameter can have one of the values:  "gzip" or "deflate".
> When asking for an attachment (GET http request), if the query parameter "compression" is found, CouchDB will send the attachment compressed to the client (and sets the header Content-Encoding with gzip or deflate).
> Further, it adds a new config option "treshold_for_chunking_comp_responses" (httpd section) that specifies an attachment length threshold. If an attachment has a length >= than this threshold, the http response will be chunked (besides compressed).
> Note that using non chunked compressed  body responses requires storing all the compressed blocks in memory and then sending each one to the client. This is a necessary "evil", as we only know the length of the compressed body after compressing all the body, and we need to set the "Content-Length" header for non chunked responses. By sending chunked responses, we can send each compressed block immediately, without accumulating all of them in memory.
> Examples:
> $ curl http://localhost:5984/testdb/testdoc1/readme.txt?compression=gzip
> $ curl http://localhost:5984/testdb/testdoc1/readme.txt?compression=deflate
> $ curl http://localhost:5984/testdb/testdoc1/readme.txt   # attachment will not be compressed
> $ curl http://localhost:5984/testdb/testdoc1/readme.txt?compression=rar   # will give a 500 error code
> Etap test case included.
> Feedback would be very welcome.
> cheers

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.