couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Joseph Davis (JIRA)" <>
Subject [jira] Commented: (COUCHDB-583) adding ?compression=(gzip|deflate) optional parameter to the attachment download API
Date Mon, 30 Nov 2009 04:26:20 GMT


Paul Joseph Davis commented on COUCHDB-583:


In a nutshell, the impedance mismatch is that webmachine wants to have control of the logic.
Ie, webmachine wants a callable that it can call at its leisure. When CouchDB writes view
output to a socket, its basically executing a lists:foldl style callback function. In terms
of webmachine, this would be akin to a streaming body where webmachine provides a fun that
CouchDB could call to write to the client socket.

The relevant code is at [1]. Its not the most centralized, but that's the fold function defined
that does the view output.

I haven't quite pieced together how webmachine is streaming to the socket. I've read the response
handling code a few times and near as I can tell its not actually streaming the body over
the socket, just consuming an iterator before charset and encoding functions are applied.

Either way, bottom line is that to fit neatly into CouchDB code, webmachine would handle a
response like {writer, fun Write/1} where the argument to the Write fun was a callable that
writes bytes to the socket. The charset and transfer encodings would then be required to be
streamable writers. Without that behavior the nearest thing I can think of would be to do
alot of message passing to invert flow control which while possible seems ungood.


> adding ?compression=(gzip|deflate) optional parameter to the attachment download API
> ------------------------------------------------------------------------------------
>                 Key: COUCHDB-583
>                 URL:
>             Project: CouchDB
>          Issue Type: New Feature
>          Components: HTTP Interface
>         Environment: CouchDB trunk revision 885240
>            Reporter: Filipe Manana
>         Attachments: jira-couchdb-583-1st-try-trunk.patch, jira-couchdb-583-2nd-try-trunk.patch
>   Original Estimate: 24h
>  Remaining Estimate: 24h
> The following new feature is added in the patch following this ticket creation.
> A new optional http query parameter "compression" is added to the attachments API.
> This parameter can have one of the values:  "gzip" or "deflate".
> When asking for an attachment (GET http request), if the query parameter "compression"
is found, CouchDB will send the attachment compressed to the client (and sets the header Content-Encoding
with gzip or deflate).
> Further, it adds a new config option "treshold_for_chunking_comp_responses" (httpd section)
that specifies an attachment length threshold. If an attachment has a length >= than this
threshold, the http response will be chunked (besides compressed).
> Note that using non chunked compressed  body responses requires storing all the compressed
blocks in memory and then sending each one to the client. This is a necessary "evil", as we
only know the length of the compressed body after compressing all the body, and we need to
set the "Content-Length" header for non chunked responses. By sending chunked responses, we
can send each compressed block immediately, without accumulating all of them in memory.
> Examples:
> $ curl http://localhost:5984/testdb/testdoc1/readme.txt?compression=gzip
> $ curl http://localhost:5984/testdb/testdoc1/readme.txt?compression=deflate
> $ curl http://localhost:5984/testdb/testdoc1/readme.txt   # attachment will not be compressed
> $ curl http://localhost:5984/testdb/testdoc1/readme.txt?compression=rar   # will give
a 500 error code
> Etap test case included.
> Feedback would be very welcome.
> cheers

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message