couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <robert.new...@gmail.com>
Subject Re: [jira] Commented: (COUCHDB-583) storing attachments in compressed form and serving them in compressed form if accepted by the client
Date Tue, 26 Jan 2010 13:51:13 GMT
that was my intention, but the option to send the encrypted bytes (for
decryption at the client end) is intriguing and also echoes the choice
to send compressed vs uncompressed responses.

I don't mean to hold up this work and I doubt I'll have a patch any
time soon, it just seems that these two features have significant
overlap (you can send data in with a transformation applied or not,
and request it with or without that transformation).

My brief look at the related code led me to believe that adding
encryption support would touch several places, and I would think that
most, perhaps all, of those places would also be touched by
compression support.

Sorry to be vague, I only intended to add another perspective to the
discussion.

On Tue, Jan 26, 2010 at 11:01 AM, Filipe David Manana
<fdmanana@gmail.com> wrote:
> Hi Robert,
>
> That's interesting.
> I think that abstraction is doable, but maybe not trivial.
>
> In your idea, you plan to always decrypt the docs/attachments before sending
> them to the client?
>
>
> On Tue, Jan 26, 2010 at 11:34 AM, Robert Newson <robert.newson@gmail.com>wrote:
>
>> fyi: I have a (much delayed) plan to work up an encryption patch for
>> documents and attachments. Since encryption and compression both apply
>> at the same level (and the order of the two is important), I wonder if
>> that would change the approach taken here? That is, would an
>> abstraction that allowed a chain of transformations when storing (and
>> the reverse chain when retrieving) be in order?
>>
>> B.
>>
>> On Tue, Jan 26, 2010 at 8:02 AM, Filipe Manana (JIRA) <jira@apache.org>
>> wrote:
>> >
>> >    [
>> https://issues.apache.org/jira/browse/COUCHDB-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804946#action_12804946]
>> >
>> > Filipe Manana commented on COUCHDB-583:
>> > ---------------------------------------
>> >
>> > @Chris
>> >
>> > Good point, I totally agree.
>> > It would be interesting to test with real couchapps, real data and see
>> how worth it really is.
>> >
>> > A 10Mb text file, for instance, was compressed to about 100Kb in one of
>> my tests.
>> >
>> > Also, as for the minified JavaScript files for example, it's still worth
>> compressing them. For example, the minified Ext JS lib file (
>> http://www.extjs.com,  ext-all.js) is about 630Kb big. Compressed with
>> gzip stays at about 170Kb, therefore a reasonably good size reduction.
>> >
>> > As Damien said in a previous comment, not only saves disk space but also
>> reduces disk IO (attachment download requests, compaction).
>> >
>> > I also look forward to see the impact on real, production level,
>> applications.
>> >
>> >> storing attachments in compressed form and serving them in compressed
>> form if accepted by the client
>> >>
>> ----------------------------------------------------------------------------------------------------
>> >>
>> >>                 Key: COUCHDB-583
>> >>                 URL: https://issues.apache.org/jira/browse/COUCHDB-583
>> >>             Project: CouchDB
>> >>          Issue Type: New Feature
>> >>          Components: Database Core, HTTP Interface
>> >>         Environment: CouchDB trunk
>> >>            Reporter: Filipe Manana
>> >>         Attachments: couchdb-583-trunk-10th-try.patch,
>> couchdb-583-trunk-11th-try.patch, couchdb-583-trunk-12th-try.patch,
>> couchdb-583-trunk-13th-try.patch, couchdb-583-trunk-14th-try-git.patch,
>> couchdb-583-trunk-15th-try-git.patch, couchdb-583-trunk-3rd-try.patch,
>> couchdb-583-trunk-4th-try-trunk.patch, couchdb-583-trunk-5th-try.patch,
>> couchdb-583-trunk-6th-try.patch, couchdb-583-trunk-7th-try.patch,
>> couchdb-583-trunk-8th-try.patch, couchdb-583-trunk-9th-try.patch,
>> jira-couchdb-583-1st-try-trunk.patch, jira-couchdb-583-2nd-try-trunk.patch
>> >>
>> >>
>> >> This feature allows Couch to gzip compress attachments as they are being
>> received and store them in compressed form.
>> >> When a client asks for downloading an attachment (e.g. GET
>> somedb/somedoc/attachment.txt), the attachment is sent in compressed form if
>> the client's http request has gzip specified as a valid transfer encoding
>> for the response (using the http header "Accept-Encoding"). Otherwise couch
>> decompresses the attachment before sending it back to the client.
>> >> Attachments are compressed only if their MIME type matches one of those
>> listed in a separate config file. Compression level is also configurable in
>> the default.ini file.
>> >> This follows Damien's suggestion from 30 November:
>> >> "Perhaps we need a separate user editable ini file to specify
>> compressable or non-compressable files (would probably be too big for the
>> regular ini file). What do other web servers do?
>> >> Also, a potential optimization is to compress the file while writing to
>> disk, and serve the compressed bytes directly to clients that can handle it,
>> and decompressed for those that can't. For compressable types, it's a win
>> for both disk IO for reads and writes, and CPU on read."
>> >> Patch attached.
>> >
>> > --
>> > This message is automatically generated by JIRA.
>> > -
>> > You can reply to this email to add a comment to the issue online.
>> >
>> >
>>
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com
> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>

Mime
View raw message