couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Borwankar <>
Subject Re: MIME dump/load and implications
Date Thu, 06 Aug 2009 23:06:30 GMT
Adam Kocoloski wrote:
> Hi Nitin, sure.  I think Paul just meant that if we wrote it for 
> replication, anyone could use the same facility to build something 
> like what you're talking about.

Hi Adam, Paul,

Yes certainly agreed in theory but in practice I see two major 
obstacles, one social and the other technical to this approach.

a) The number of people who can work on the replication code is a 
handful and this hugely raises the bar and puts all this in the domain 
of "someday-maybe" - "it will need to get on the release schedule" etc. 
which is less attractive to me.

On the other hand doing it in userland via couchapp throws open the 
gates for anyone to start experimenting with this and then the best 
stuff will emerge and get traction, one hopes.  I couldn't possibly make 
any useful contribution at the replication level at my current level of 
knowledge of Couch/Erlang/... but hacking at the couchapp level is not 
out of the range of my capabilities.  I suspect most users of Couch are 
in the same boat as me.

So I deliberately focus on a REST based MIME-payload protocol that could 
run in parallel with the REST based JSON-payload protocol.
At this level of abstraction it is easy to understand and talk about for 
a much huger audience.  Moreover the amount of work needed to bridge the 
gap between the already working python code and a couchapp/javascript 
impl of this is much smaller.

b)  This will all also be possible in the replication layer but a 
messaging system has additional requirements such as keeping cumulative 
track of where the message came from - adding a MIME-header (JSON 
attribute)  i.e modifying the doc at each hop etc.  These concerns are 
not fundamental to point-point doc replication and need to be layered on 
top/in addition to replication.  Replication needs to faithfully 
reproduce a doc from one endpoint to another without the kinds of 
modification that happen to headers (metadata attributes) in message 

So thinking of messaging as "merely" replication glosses over some 
fundamental issues that distinguish message documents in motion, from 
more general documents.  Message documents have container level 
attributes that get modified during the process of store-and-forward to 
track the motion of the doc.
Generic documents in CouchDB have no such requirement and it may be 
counterproductive to intermingle the requirements.

In fact it may be possible to make a stronger statement - Couch 
replication should focus single mindedly on replication of documents in 
a pt-to-pt fashion.
Messaging semantics can then be layered on top of replication, in an 
upper layer if desired - or not.  Injecting messaging concerns 
(especially modification of container attributes) into replication is a 
distraction of focus for replication, IMHO and possibly counter to the 
requirements of faithful replication.

Hope this clarifies why I reacted somewhat strongly to conflating REST 
based messaging with replication - they overlap but should not be 
equated if we want to build a messaging system, != a peer to peer doc 
sharing system.

> After all, the replicator just speaks to CouchDB servers using the 
> same HTTP requests as everyone else.
> I'm sure jchris knows better whether mimeparse would be suitable for 
> this.  Cheers,

Mimeparse.js - looking at the code - seems focused on making the best 
match of content-type to provide given an accept header so it may not be 
the best match for this.



> Adam
> On Aug 6, 2009, at 5:26 PM, Nitin Borwankar wrote:
>> Hi Paul,
>> I never used the word replication - it should be possible to create a 
>> REST based couchapp driven MIME transfer p2p web quiteindependent of 
>> replication which is also cool.  Front the couchapp with the usual 
>> auth-proxy stuff for now so only auth'ed people can communicate with 
>> you.
>> Just replace JSON with MIME in all the reference docs and make the 
>> URL's point to a design doc that does the transformations.
>> On the way out it could be just _shows or an _list that takes 
>> multiple objects and wraps them as a mime multipart.
>> On the way in set up some REST endpoints that take POST's, parse mime 
>> multiparts ( jchris's mimeparser?)  convert to Couch docs, manage 
>> attachments and puts them in _attachments ... and we're off to the 
>> races - free user controlled, MIME-and-Mail-as-aplatform driven 
>> webmail apps for all.
>> Yay Couch!
>> Nitin
>> Paul Davis wrote:
>>> Definitely some interesting points here. There have been discussions
>>> on using multipart-mime messaging in the replication protocol which
>>> could setup for some interesting prospects like this. I'm not sure on
>>> specifics in terms of replication, but having an endpoint that allows
>>> edits via multipart-mime could be a very fun thing to play with.
>>> Also, AFAIK there's nothing that prevents an isomorphic
>>> representation. As you point out, couchdb-python handles everything
>>> just fine here.
>>> Paul Davis
>>> On Thu, Aug 6, 2009 at 4:46 PM, Nitin Borwankar<> 
>>> wrote:
>>>> Hi guys,
>>>> I see that the python based dump/load uses MIME multipart docs as 
>>>> an on-disk
>>>> serialisation format for couchdb databases.
>>>> An overall question then arises - can CouchDB be considered a MIME 
>>>> database
>>>> which oh also happens to talk JSON?
>>>> So before that - is there a 1-1 strong correspondence between a 
>>>> CouchDB
>>>> document and a MIME multipart, or are there things around the edges 
>>>> that are
>>>> crufty - I would assume a strong correspondence since dump/load 
>>>> uses it and
>>>> I haven't seen any caveast about document content that is not 
>>>> dumpable.
>>>> So assuming the 1-1 correspondence - could one use some 
>>>> "translation layer"
>>>> couchapp that accepts arbitrary content/type + multipart-mixed MIME 
>>>> object
>>>> over HTTP and then transparenty serialise them to JSON underneath.
>>>> Given that dump/load already does this - it would see that there 
>>>> are no
>>>> obvious glaring flaws in this logic - but I have been known to be 
>>>> wrong,
>>>> once :-).
>>>> If this is indeed feasible - then each CouchDB + MIME-trans becomes 
>>>> a web
>>>> mail node - and Couch begins to be the platform for  a messaging 
>>>> revolution
>>>> as well as an application revolution. I am thinking now not as 
>>>> CouchDB for
>>>> backing up your email - but CouchDB as your mail client/server for 
>>>> p2p MIME
>>>> based "email".
>>>> Permissions etc are important to avoid complete disaster of course 
>>>> - but
>>>> private high quality communication that just reuses existing message
>>>> formats, with better storage and transport would seem like an idea 
>>>> whose
>>>> time has come a long time ago and has been knocking at the door for a
>>>> decade.
>>>> Yes, yes, there's the issue of spam - so see the P.S.
>>>> Just a few idle thoughts,
>>>> Nitin
>>>> P.S.  Back in 1998 I tried to convince Sybase to have MIME as a 
>>>> native type
>>>> in the db and it even got speced out ( I have the spec with the 
>>>> date on it!
>>>> ) but got canned becous ethe VP of enginnering wanted to know "what 
>>>> was the
>>>> market exactly for this kind of stuff".  Other than that I was 
>>>> granted  a
>>>> patent for doing p2p discussions over email back in 2003 - I let it 
>>>> expire
>>>> for multiple reasons.  So I am somewhat non-naive about and aware 
>>>> of the
>>>> issues and pitfalls around this sort of thinking. At the same time 
>>>> I am of
>>>> the strong belief that when one looks at messages as data to be 
>>>> moved around
>>>> between endpoints with well defined addressing schemes, and one 
>>>> ignores the
>>>> protocols for a bit, then all sorts of fun things start to happen.
>>>> 37% of all statistics are made up on the spot
>>>> -------------------------------------------------------------------------------------

>>>> Nitin Borwankar

View raw message