couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benoit Chesneau <>
Subject Re: mem3 and forced db fragmentation?
Date Mon, 03 Mar 2014 04:30:00 GMT
On Mon, Mar 3, 2014 at 3:26 AM, Adam Kocoloski <> wrote:
> On Feb 28, 2014, at 4:31 AM, Benoit Chesneau <> wrote:
>> Looking at the code of mem3 it seems that all databases are oblgatory
>> fragmented (sharded). I understand that you can have only 1 shard /
>> database but then it will be still stored at
>> `<database_dir>/shard/<dbname>`.
>> Is there any plan of having non fragmented databases stored in their
>> own place ie `<database_dir>/<dbname>`  ? Would be interesting when
>> you want to manage different policies of backup depending on the type
>> of database (fragmented or not). Which place should I look to make it
>> happen?
>> - benoit
> I'm not entirely certain I understand what you're looking for there Benoit.  I classify
databases in two categories: "clustered" and "local".

> A clustered database is available via the clustered HTTP interface and has its data stored
in files inside <database_dir>/shards/<range>/<dbname>.  This is true even
for a Q=1,N=1 database where this is only one file for the database in the entire cluster.
> On the other hand, a local database exists on a specific node in the cluster, and is
not accessible via the clustered HTTP interface.  The data for this database is stored at
> What exactly did you mean by a non-fragmented database?  It sort of sounds like you're
wanting to conflate a Q=1,N=1 clustered database with a local database, but those are very
different concepts (not least because multiple nodes in a cluster can have a local database
with the same name but different content).  I can honestly say we've never considered storing
the files for Q=1 or Q=1,N=1 databases somewhere different than other clustered databases.

A fragmented or sharded database is the clustered one. I call them
fragmented databases since afaik a clustered database is spitted in
multiple pieces.

> Can you say more about your motivations here?  You talked about backup policies; while
I could absolutely see wanting to implement a different backup policy for N=1 databases it's
not obvious to me why you'd want to treat Q=1 and Q>1 databases separately.

Having them in there own folder instead of shard them them easily more
 distinct on the fs than having them in shards/ and i could easily
find db.couch by its name without using the API on the disk.

Also I am wondering right now how the transition from a the current
couchdb to the new one can be done. It would be quite easier imo to
have 1 HTTP layer and flag the  "special databases" to not expose them
in the HTTP API except for the admins.

- benoit

View raw message