couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <>
Subject Re: continuous replication API discussion
Date Mon, 17 Aug 2009 03:48:11 GMT
On Aug 12, 2009, at 12:41 AM, Chris Anderson wrote:

> I think a view of _local docs would be ideal. I've been wishing for
> this or a long time, glad to see we're getting close to it.

I was thinking just a single _local_docs view, but it sounds like you  
might have grander plans.  I assume any view on _local docs will have  
to a built-in one.

>> * we could automatically add a field to the config document that  
>> identifies
>> the corresponding _local doc ID.  We'd have to use something like  
>> the new
>> _update handler to validate this field on every update, as a change  
>> in
>> authentication information means a change in the corresponding  
>> _local doc
>> ID.
> I think trying to keep them in sync is a smell. The replication
> manager can perhaps just use a group view of replication history to
> list all the potential replication targets.

I don't think it's so bad.  We can use the same function that  
generates the replication ID from the POST body to add a _local_id  
reserved field to the doc in the _replication DB.  It's just like a  

>> A thought -- what if one-off POSTs to /_replicate automatically  
>> saved a
>> config document as well, but with a "continuous":false field?  Then  
>> it would
>> be easy to view all the replications that ever originated on the  
>> server, and
>> admins could make certain ones continuous just by flipping a bit.

> What kind of _local doc artifacts does continuous replication leave in
> the current implementation - maybe a _local view can give us
> everything we need.

I'm worried about saving source and target values with authentication  
information in _local docs that are world readable and saved on both  
servers.  It would be nice to see some info on replications triggered  
from other servers, though, and viewing _local docs allows for that.   
Anyway, to answer your question ...

CouchDB deterministically generates a replication ID from the server  
hostname and the source and target values in the body of the POST.  It  
saves a _local document with this ID on both source and target that  
looks like

   "session_id": uuid(),
   "source_last_seq": integer(),
   "history": array()

The history is capped at 50 entries, each of which looks like

   "session_id": uuid(), % for the head of the list this is the same  
as above
   "recorded_seq": integer(), % for the head this is the same as  
   "start_time": rfc1123_date(),
   "end_time": rfc1123_date(),
   "start_last_seq": integer(),
   "end_last_seq": integer(), % might be different from recorded_seq  
if a server restarted during replication
   "missing_checked": integer(),
   "missing_found": integer(),
   "docs_read": integer(),
   "docs_written": integer(),
   "doc_write_failures": integer()

On Aug 12, 2009, at 5:08 AM, Simon Metson wrote:

> Why not have a reserved _named database for each configuration  
> section? Depending on what the _config eventually evolves to hold  
> you might find it getting hit frequently - you don't want your  
> replication settings getting screwed up by settings for other  
> components. You also may want replication settings to be "public"  
> but other parts to be top secret, I'd say that's easier to manage at  
> a per database level than not.

I wouldn't completely replace the current .ini configuration files  
with a database -- I think there's real value in being able to change  
some settings without the server running.

> Given that how about _replication for a name?
> Cheers
> Simon

I like it.

On Aug 12, 2009, at 10:45 AM, Randall Leeds wrote:

> fwiw, I can see it being very useful to be able to replicate a _config
> database or a _replication database.

Just thinking about all the implications makes me nervous!  I think  
managing server configurations is probably best left to configuration  
management tools like Opscode Chef and its ilk.  I could be wrong  
though.  If we wanted to support this I think it would be easy enough  
to do for _replication.  Obviously for _config we'd need something  
like what Simon proposed.


I'll push forward on the _replication DB for configuring continuous  
replications, but I'm still not sure of the best path forward on  
viewing/managing one-off replications.  I'm torn between auto-saving a  
document into the _replication DB, adding additional info to the  
_local docs, or doing both (full info in the _replication DB, "public"  
info in the _local doc).  Best,


View raw message