Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of kenneth.kalmer@gmail.com
 designates 209.85.219.18 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Mx1ryK1T7KyfObPV1962vo439no+nKsAme8yWuJs83dHdJbOrXUjSJ50TV38o3vxDQ
         0l8EjkGJONDZ5Oykw1v/t0gayzIMjAVTs+E/7F2ZT6EsxSTn6iE78i/8QrNpIWIC+0bT
         KHbpXbhba8mrtpB9bXvOVp1yKfQR7JRUPcAkM=
MIME-Version: 1.0
In-Reply-To: <E27245D3-DD9A-453F-AA78-013A0F1CCC5F@apache.org>
References: <fad9d4840902120613m32a79877gf0503bda1f1fc27d@mail.gmail.com>
	 <20090212161918.GB24119@uk.tiscali.com>
	 <E27245D3-DD9A-453F-AA78-013A0F1CCC5F@apache.org>
Date: Thu, 12 Feb 2009 22:38:23 +0200
Message-ID: <fad9d4840902121238u341c3b1bp1c8b40fa388a9b1c@mail.gmail.com>
Subject: Re: Couch as a mail store?
From: Kenneth Kalmer <kenneth.kalmer@gmail.com>
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=001517478a2ea03a820462beb315

--001517478a2ea03a820462beb315
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On Thu, Feb 12, 2009 at 7:24 PM, Jan Lehnardt <jan@apache.org> wrote:

>
> On 12 Feb 2009, at 17:19, Brian Candler wrote:
>
>  Of course, SMTP clients don't mind a small delay before they get their 250
>> OK at the end of the message; you can therefore write a number of batches
>> and do an fsync() every second or so, as long as you remember not to send
>> the acknowledgement back to each client until *after* the fsync has
>> completed.
>>
>
> CouchDB does the fsync()-a-second currently.
>

Not that it matters though: Postfix (the only MTA I have experience with,
and years at it) spools the mail first into one of several queues. Once in
the queue it send the 250 back and the client is freed. The same should
happen before hitting couch, at least in my head. This also allows for
temporary failures between couch and the MTA without affecting the client.

But absolutely guaranteeing delivery is impossible, you can aim for 5 9's,
or 7 9's even, but all 100% won't happen...


 (2) Couch won't let you write a document to disk in chunks; if it doesn't
> get an up-front Content-Length: header then it will buffer the whole thing
> in RAM.
>
> So if you receive very large E-mail messages, you may wish to buffer them
> locally (e.g. in a tempfile on another disk) before sending them as a
> single
> document to Couch.
>
> Note that you sometimes get an indication of the message size in a SMTP
> transaction, but it's not guaranteed to be accurate; so you won't know the
> true size until you've read it in.
>

Sucky thing with email...

A patch to allow for streaming unknown-length attachments into CouchDB
> is in the works.
>

If you have a store-and-forward MTA you could live without this one, but it
sounds like a handy addition.

 (3) As you say, messages are stored and deleted frequently. You may end up
> having to compact your message store frequently, which means basically
> reading the whole store from start to end and rewriting it to a new file.
> This has to be done when write load isn't too high, to ensure that it will
> complete.
>

CouchDB's roadmap includes an item about pausing writes to allow compaction
> to catch up on high write loads.
>

Also nice, but again a store-and-forward MTA would help alleviate this
issue. It would be even better if the MTA could be made aware of the
compaction process and slow down ingestion.

Thanks for all the feedback so far, sounds like I've hit a common nail right
on the head here.


-- 
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za

--001517478a2ea03a820462beb315--