The basic intention of my patch would be to ensure that unencrypted
documents and attachments are not on disk. You're right that there are
many other questions to answer for a general encryption feature.
For my exact case, which might always remain a local patch, I wouldn't
mind the data passing unencrypted to and from the boxes, but should be
encrypted while "at rest" on disk.
B.
On Tue, Jan 26, 2010 at 2:02 PM, Filipe David Manana <fdmanana@gmail.com> wrote:
> Robert,
>
> I think your plans are very interesting, as they present not only
> interesting challenges but the feature itself I find useful also.
> Questions such as:
> - what kind of encryption? (symmetric, asymmetric, or both)
> - where are the keys stored? Or for the symmetric case, we would use a
> diffie-helman protocol for e.g.?
> - is the objective to have privacy at the DB storage level or also at the
> network level? (and force the decryption on the client side only)
> there are many more details of course.
> I do have the same opinion as you, the code would affect many of the parts
> regarding the compression (specially couch_stream). For doc compression, I
> imagine it would touch more places, and also present some difficulties to
> assure compatibility with the previous DB file formats.
> Let me know if somehow I can help you.
> cheers
>
> On Tue, Jan 26, 2010 at 2:51 PM, Robert Newson <robert.newson@gmail.com>
> wrote:
>>
>> that was my intention, but the option to send the encrypted bytes (for
>> decryption at the client end) is intriguing and also echoes the choice
>> to send compressed vs uncompressed responses.
>>
>> I don't mean to hold up this work and I doubt I'll have a patch any
>> time soon, it just seems that these two features have significant
>> overlap (you can send data in with a transformation applied or not,
>> and request it with or without that transformation).
>>
>> My brief look at the related code led me to believe that adding
>> encryption support would touch several places, and I would think that
>> most, perhaps all, of those places would also be touched by
>> compression support.
>>
>> Sorry to be vague, I only intended to add another perspective to the
>> discussion.
>>
>> On Tue, Jan 26, 2010 at 11:01 AM, Filipe David Manana
>> <fdmanana@gmail.com> wrote:
>> > Hi Robert,
>> >
>> > That's interesting.
>> > I think that abstraction is doable, but maybe not trivial.
>> >
>> > In your idea, you plan to always decrypt the docs/attachments before
>> > sending
>> > them to the client?
>> >
>> >
>> > On Tue, Jan 26, 2010 at 11:34 AM, Robert Newson
>> > <robert.newson@gmail.com>wrote:
>> >
>> >> fyi: I have a (much delayed) plan to work up an encryption patch for
>> >> documents and attachments. Since encryption and compression both apply
>> >> at the same level (and the order of the two is important), I wonder if
>> >> that would change the approach taken here? That is, would an
>> >> abstraction that allowed a chain of transformations when storing (and
>> >> the reverse chain when retrieving) be in order?
>> >>
>> >> B.
>> >>
>> >> On Tue, Jan 26, 2010 at 8:02 AM, Filipe Manana (JIRA) <jira@apache.org>
>> >> wrote:
>> >> >
>> >> > [
>> >>
>> >> https://issues.apache.org/jira/browse/COUCHDB-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804946#action_12804946]
>> >> >
>> >> > Filipe Manana commented on COUCHDB-583:
>> >> > ---------------------------------------
>> >> >
>> >> > @Chris
>> >> >
>> >> > Good point, I totally agree.
>> >> > It would be interesting to test with real couchapps, real data and
>> >> > see
>> >> how worth it really is.
>> >> >
>> >> > A 10Mb text file, for instance, was compressed to about 100Kb in one
>> >> > of
>> >> my tests.
>> >> >
>> >> > Also, as for the minified JavaScript files for example, it's still
>> >> > worth
>> >> compressing them. For example, the minified Ext JS lib file (
>> >> http://www.extjs.com, ext-all.js) is about 630Kb big. Compressed with
>> >> gzip stays at about 170Kb, therefore a reasonably good size reduction.
>> >> >
>> >> > As Damien said in a previous comment, not only saves disk space but
>> >> > also
>> >> reduces disk IO (attachment download requests, compaction).
>> >> >
>> >> > I also look forward to see the impact on real, production level,
>> >> applications.
>> >> >
>> >> >> storing attachments in compressed form and serving them in
>> >> >> compressed
>> >> form if accepted by the client
>> >> >>
>> >>
>> >> ----------------------------------------------------------------------------------------------------
>> >> >>
>> >> >> Key: COUCHDB-583
>> >> >> URL:
>> >> >> https://issues.apache.org/jira/browse/COUCHDB-583
>> >> >> Project: CouchDB
>> >> >> Issue Type: New Feature
>> >> >> Components: Database Core, HTTP Interface
>> >> >> Environment: CouchDB trunk
>> >> >> Reporter: Filipe Manana
>> >> >> Attachments: couchdb-583-trunk-10th-try.patch,
>> >> couchdb-583-trunk-11th-try.patch, couchdb-583-trunk-12th-try.patch,
>> >> couchdb-583-trunk-13th-try.patch, couchdb-583-trunk-14th-try-git.patch,
>> >> couchdb-583-trunk-15th-try-git.patch, couchdb-583-trunk-3rd-try.patch,
>> >> couchdb-583-trunk-4th-try-trunk.patch, couchdb-583-trunk-5th-try.patch,
>> >> couchdb-583-trunk-6th-try.patch, couchdb-583-trunk-7th-try.patch,
>> >> couchdb-583-trunk-8th-try.patch, couchdb-583-trunk-9th-try.patch,
>> >> jira-couchdb-583-1st-try-trunk.patch,
>> >> jira-couchdb-583-2nd-try-trunk.patch
>> >> >>
>> >> >>
>> >> >> This feature allows Couch to gzip compress attachments as they
are
>> >> >> being
>> >> received and store them in compressed form.
>> >> >> When a client asks for downloading an attachment (e.g. GET
>> >> somedb/somedoc/attachment.txt), the attachment is sent in compressed
>> >> form if
>> >> the client's http request has gzip specified as a valid transfer
>> >> encoding
>> >> for the response (using the http header "Accept-Encoding"). Otherwise
>> >> couch
>> >> decompresses the attachment before sending it back to the client.
>> >> >> Attachments are compressed only if their MIME type matches one
of
>> >> >> those
>> >> listed in a separate config file. Compression level is also
>> >> configurable in
>> >> the default.ini file.
>> >> >> This follows Damien's suggestion from 30 November:
>> >> >> "Perhaps we need a separate user editable ini file to specify
>> >> compressable or non-compressable files (would probably be too big for
>> >> the
>> >> regular ini file). What do other web servers do?
>> >> >> Also, a potential optimization is to compress the file while writing
>> >> >> to
>> >> disk, and serve the compressed bytes directly to clients that can
>> >> handle it,
>> >> and decompressed for those that can't. For compressable types, it's a
>> >> win
>> >> for both disk IO for reads and writes, and CPU on read."
>> >> >> Patch attached.
>> >> >
>> >> > --
>> >> > This message is automatically generated by JIRA.
>> >> > -
>> >> > You can reply to this email to add a comment to the issue online.
>> >> >
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Filipe David Manana,
>> > fdmanana@gmail.com
>> > PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>> >
>> > "Reasonable men adapt themselves to the world.
>> > Unreasonable men adapt the world to themselves.
>> > That's why all progress depends on unreasonable men."
>> >
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com
> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
>
|