couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: mem3 and forced db fragmentation?
Date Tue, 04 Mar 2014 01:43:35 GMT
Hi Benoit, it's important to be precise.  I tried to lock down what you meant by a non-fragmented
database but I'm afraid we're not there yet.  In this reply you say 

> A fragmented or sharded database is the clustered one

which I'd read as saying that as a Q=1,N=1 database still counts as a fragmented or sharded
database, but I don't think that's what you meant.  I'm honestly not sure which specific databases
you want to move out of shards/ and into the top-level.

> Also I am wondering right now how the transition from a the current
> couchdb to the new one can be done. It would be quite easier imo to
> have 1 HTTP layer and flag the  "special databases" to not expose them
> in the HTTP API except for the admins.

Now that's an interesting discussion.  We've mulled it over internally at Cloudant a bit in
the past but never really came to a conclusion.  I don't find the :5984 / :5986 split elegant,
but I'll also say that getting the API for local vs. clustered databases exactly right is
not at the top of my priority list.  In fact I think it may be the sort of thing that we could
defer until after the first release that includes clustering capabilities.

Adam
 
On Mar 2, 2014, at 11:30 PM, Benoit Chesneau <bchesneau@gmail.com> wrote:

> On Mon, Mar 3, 2014 at 3:26 AM, Adam Kocoloski <kocolosk@apache.org> wrote:
>> On Feb 28, 2014, at 4:31 AM, Benoit Chesneau <bchesneau@gmail.com> wrote:
>> 
>>> Looking at the code of mem3 it seems that all databases are oblgatory
>>> fragmented (sharded). I understand that you can have only 1 shard /
>>> database but then it will be still stored at
>>> `<database_dir>/shard/<dbname>`.
>>> 
>>> Is there any plan of having non fragmented databases stored in their
>>> own place ie `<database_dir>/<dbname>`  ? Would be interesting when
>>> you want to manage different policies of backup depending on the type
>>> of database (fragmented or not). Which place should I look to make it
>>> happen?
>>> 
>>> - benoit
>> 
>> I'm not entirely certain I understand what you're looking for there Benoit.  I classify
databases in two categories: "clustered" and "local".
> 
>> 
>> A clustered database is available via the clustered HTTP interface and has its data
stored in files inside <database_dir>/shards/<range>/<dbname>.  This is
true even for a Q=1,N=1 database where this is only one file for the database in the entire
cluster.
>> 
>> On the other hand, a local database exists on a specific node in the cluster, and
is not accessible via the clustered HTTP interface.  The data for this database is stored
at <database_dir>/<dbname>.
>> 
>> What exactly did you mean by a non-fragmented database?  It sort of sounds like you're
wanting to conflate a Q=1,N=1 clustered database with a local database, but those are very
different concepts (not least because multiple nodes in a cluster can have a local database
with the same name but different content).  I can honestly say we've never considered storing
the files for Q=1 or Q=1,N=1 databases somewhere different than other clustered databases.
> 
> A fragmented or sharded database is the clustered one. I call them
> fragmented databases since afaik a clustered database is spitted in
> multiple pieces.
> 
>> 
>> Can you say more about your motivations here?  You talked about backup policies;
while I could absolutely see wanting to implement a different backup policy for N=1 databases
it's not obvious to me why you'd want to treat Q=1 and Q>1 databases separately.
> 
> Having them in there own folder instead of shard them them easily more
> distinct on the fs than having them in shards/ and i could easily
> find db.couch by its name without using the API on the disk.
> 
> Also I am wondering right now how the transition from a the current
> couchdb to the new one can be done. It would be quite easier imo to
> have 1 HTTP layer and flag the  "special databases" to not expose them
> in the HTTP API except for the admins.
> 
> - benoit


Mime
View raw message