couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipe David Manana <fdman...@apache.org>
Subject Re: all_dbs_active error and continuous replications
Date Tue, 06 Dec 2011 09:52:10 GMT
On Tue, Dec 6, 2011 at 9:29 AM, Gregor Martynus <gregor@martynus.net> wrote:
> What I really don't understand is, _why_ the continuous replication keep
> the db files open? I'm not an Erlang developer at all, sorry if this is a
> dump question/thought, but shouldn't the continuous replication be
> triggered only if a change happens in the db, instead of keeping it alive
> all the time? Is this a bug or probably something that might be improved in
> future?

Actually it doesn't keep the source and database open when there's no
changes to transfer from the source to the target.
What happens is that when started, the replication records the
databases' instance start times - they're timestamps that tell when
the databases where open by the couchdb server.

Whenever the replication commits a checkpoint (it is committed against
the source and the target), it verifies if the endpoints' current
instance start times are the same as when it started. If they're not
the same, then you'll get that max_dbs_open error message from the
replicator.

When the system has max_dbs_open and a request to open another
database X comes, it will close the least recently used database Y in
order to open the new one. If later on a request to open the database
X that was previously closed (because we reached max_dbs_open)
arrives, the new least recently used database will be closed, and X
will be reopened - at this point it gets a new instance start
timestamp.

The reason to abort if the instance start timestamps changed, it to
prevent you from getting into an inconsistent state. For example,
during an idle period (when there's no changes to transfer from source
to target) someone could swap database files - if the replicator
ignored the timestamps, it could mean in the end your target would be
missing all the changes transferred before the file swap.

One thing I have been thinking for sometime is to have the server
remember the timestamps and header md5 of each database opened since
it started. When it reopens a database that was previously closed by
the LRU system (due to max_dbs_open), it would use the cached
timestamp if the database's header md5 is the same as the one that's
cached along with the timestamp.
I haven't convinced myself this is a flawless/good idea however and
never made extensive tests.


>
> Thanks
>
> × Gregor
>
> On Tue, Dec 6, 2011 at 10:16 AM, Benoit Chesneau <bchesneau@gmail.com>wrote:
>
>> On Tue, Dec 6, 2011 at 10:11 AM, Gregor Martynus <gregor@martynus.net>
>> wrote:
>> > Thanks Benoit, I know about this setting. But at some point, this won't
>> > work anymore. My question is if there is any way to stop continuous
>> > replications to keep the db files active changes have been replicated?
>> Does
>> > that make sense?
>> >
>> >
>>
>> other way would be to queue changes events and when replicated dbs <
>> max dbs open a new task if a change happened on this . Something like
>> it.
>>
>> Otherwise to answer to your question, if you use the _replicator db
>> you can interrupt a task by just deleting the doc.
>>
>> - benoît
>>



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Mime
View raw message