couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Filipe David Manana <fdman...@apache.org>
Subject Re: [jira] [Commented] (COUCHDB-1153) Database and view index compaction daemon
Date Tue, 16 Aug 2011 21:23:42 GMT
On Tue, Aug 16, 2011 at 2:58 AM, Benoit Chesneau <bchesneau@gmail.com> wrote:
>
> Could be a local docs, But why didn't we took this path for this
> "_security" object ? Also since they are really "meta" informations,
> i've the feeling it should be solved as a special member in the db
> file, just like the "_security" object.

I don't know why _security is like it is now, that predates me, and
it's another topic :)

>
> Anyway what I really dislike is saving per db configuration in an ini
> file. Per db configuration should be done on the db. What if you more
> than 100 dbs. Having 100 lines in an ini file to parse is awkward.

I don't think the common case is to have a separate compact config for
every single database.
The fragmentation parameter, which is likely the most useful, you're
likely to not set a different value for 100 databases (neither the
period for e.g.).

For other things like the oauth tokens/secrets, the .ini system
doesn't scale. But that's again another topic.

> This is just as simple as this line, creating a db create an entry in
> a db index (or db file) that you can use later.
>
>> I suspect what you think is something like rather than scanning
>> periodically, to let the daemon be notified when a db (or view) can be
>> compacted?
>> At some point I considered reacting to db_updated events but this was
>> pretty much flooding the the event handler (daemon).
>> Was this your idea?
>>
>
> Using db events is my idea yes.  If t actually flood the db event
> handler (not sure why), then maybe we should fix it first?

The problem is when you have many dbs in the system and under a
reasonable write load, the daemon (which is the receiver of db_updated
events) receives too many messages. To know if you need to compact the
db after such message, you need to open it, and opening it on every
message is a big burden as well.
I tried this on a system with 1024 databases being updated constantly.
It also doesn't deal with the case on startup where if a db with a
high fragmentation is not updated for a long period, it won't have
compaction started.

If someone can measure the current solution's impact and present
another working alternative with a lower impact (and practical tests,
not just theory) I would be the first one wanting to make the change
asap.

>
> - benoit
>



-- 
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Mime
View raw message