Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of kenneth.kalmer@gmail.com
 designates 64.233.182.184 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=WmNqyBDuqumyWD5djC3PhML7QPiQFbtUzWvQqhDpncTIESibT+ZuihdIdli3Gfud0X
         ZL6jAGfsRnwYCgjLurKWW5RXZ0PI8q0Tn+WeOpWv103GAZOgfaZ/iXdmw/B04rdPWyXo
         2FzzxYr56esJUClcXyYyin35gSBHvePqvB6+8=
MIME-Version: 1.0
In-Reply-To: <4FE341EC-2356-4D03-A07D-C9001D2E5CCC@apache.org>
References: <fad9d4840902120613m32a79877gf0503bda1f1fc27d@mail.gmail.com>
	 <4FE341EC-2356-4D03-A07D-C9001D2E5CCC@apache.org>
Date: Thu, 12 Feb 2009 17:51:42 +0200
Message-ID: <fad9d4840902120751k652056c0xb8b627719eba3a5a@mail.gmail.com>
Subject: Re: Couch as a mail store?
From: Kenneth Kalmer <kenneth.kalmer@gmail.com>
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=00151749f7ce58e5250462bab24c

--00151749f7ce58e5250462bab24c
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On Thu, Feb 12, 2009 at 4:26 PM, Jan Lehnardt <jan@apache.org> wrote:

>
> On 12 Feb 2009, at 15:13, Kenneth Kalmer wrote:
>
>  Hi everyone
>>
>> This is my first post, please be gentle as I risk ridicule. I've been
>> lurking here for several months now, learning from others. Disclaimer, I
>> have yet do do more with couch than updating and running the tests.
>>
>
> No worries, we don't bite (...usually :).
>

I've noticed :)

 How would couch fair as a backend for a mail delivery system (in concept)?
>

Two words: Perfect match.
>

My reason for persuing the concept.

 Considering you need high availability and very fast IO. Documents (email
> messages) will be created and deleted very often, some almost
> instantaneously.
>
> Couch has some great attributes that makes it sound worth exploring
> further:
>
> * Fast lookup of documents
> * Awesome replication for business continuity (especially in a low-latency
> environment like GIG-E)
> * Scales horizontally
> * Ability to pull entire mailbox for user as one result, or at least bundle
> X emails together in one response
>
> I can't recall seeing any thread on here in recent history discussing high
> document deletion rates, which is effectively the case when people pop
> their
> mail.
>

A deletion is effectively a set-deleted-flag operation. Compaction then
> takes
> care of getting rid of the file.
>

So taken from other threads, you're effectively tasked with running
compaction outside of your peak time. This is a no brainer if the other
benefits are in reachable.

While I'm here, can the docs still be recovered before compaction? Why I ask
is that it would be a bonus to be able to access the mails and do some
statistical reporting before compacting the database, if not, no issues.
Mail server admins (and those footing the bills) love excessive reporting...


 Normal filesystem-based storage of mail has other issues:
>
> * Messages often smaller than ethernet jumbo frames, so limited throughput
> (couch can overcome this by bundling messages in a single response)
> * Mostly limited by disk IO and clever tricks around solid state drive
> usage
> or stripping excessively fast disks
>
> Lets assume nothing about existing mail stores, except that filesystem ones
> don't scale will, and I don't even want to consider the possibility of
> raping an RDBMS for this.
>
> Everything is exploratory, the thought just crept into my mind a couple of
> days ago and I'd like to bounce the idea around with everyone for fun.
>
> Thanks for all the hard work, and everyones patience with newbies and
> attackers alike.
>

Hey, thanks for the nice words :)
>
> Hmm, not too much information. Let's see, if you have any more specific
> questions, just send a follow up :)
>

Well, lets try and keep this as close as couch as we can and not wander off
into the nasty world of email systems (except for effectively CRUD-ing
messages).

So mail arrives at our SMTP server. What would give us the best performance
for ingesting mail, directly writing each doc as it arrives, or having small
queues that empty out every X messages / Y seconds (whichever comes first)?
Considering one of our mail clouds does about 15GB an hour during office
hours. I know this size isn't anything when you consider larger providers,
but we're growing constantly and some time in the future we're gonna have to
become creative in how we store mail.

Retrieving mail also becomes interesting, we can use one view to get the
total number of messages for the mailbox, and then another (with parameters)
to batch them from couchdb as we deliver them to the client. Would bulk
updates here be the cheapest way of "mark all as read" or "delete", or would
you again handle documents individually?

Best


-- 
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za

--00151749f7ce58e5250462bab24c--