Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of adam.kocoloski@gmail.com
 designates 209.85.192.46 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
Subject: Re: mem3 and forced db fragmentation?
From: Adam Kocoloski <kocolosk@apache.org>
In-Reply-To: 
 <CAJNb-9qi3PAy5YYnaYGkTh=mw05bbS00J8PhBNfzXVz=t976dA@mail.gmail.com>
Date: Mon, 3 Mar 2014 20:43:35 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <DE1D9E7B-262A-4327-802E-DC74B76E19DD@apache.org>
References: 
 <CAJNb-9pg+DeDMk+fPxbtmpdnBshvUWM7CsvvCM7gjwdfC=qtrg@mail.gmail.com>
 <504DC2A5-AE9C-40AE-A82D-2F2DB983A72A@apache.org>
 <CAJNb-9qi3PAy5YYnaYGkTh=mw05bbS00J8PhBNfzXVz=t976dA@mail.gmail.com>
To: dev@couchdb.apache.org

Hi Benoit, it's important to be precise.  I tried to lock down what you =
meant by a non-fragmented database but I'm afraid we're not there yet.  =
In this reply you say=20

> A fragmented or sharded database is the clustered one

which I'd read as saying that as a Q=3D1,N=3D1 database still counts as =
a fragmented or sharded database, but I don't think that's what you =
meant.  I'm honestly not sure which specific databases you want to move =
out of shards/ and into the top-level.

> Also I am wondering right now how the transition from a the current
> couchdb to the new one can be done. It would be quite easier imo to
> have 1 HTTP layer and flag the  "special databases" to not expose them
> in the HTTP API except for the admins.

Now that's an interesting discussion.  We've mulled it over internally =
at Cloudant a bit in the past but never really came to a conclusion.  I =
don't find the :5984 / :5986 split elegant, but I'll also say that =
getting the API for local vs. clustered databases exactly right is not =
at the top of my priority list.  In fact I think it may be the sort of =
thing that we could defer until after the first release that includes =
clustering capabilities.

Adam
=20
On Mar 2, 2014, at 11:30 PM, Benoit Chesneau <bchesneau@gmail.com> =
wrote:

> On Mon, Mar 3, 2014 at 3:26 AM, Adam Kocoloski <kocolosk@apache.org> =
wrote:
>> On Feb 28, 2014, at 4:31 AM, Benoit Chesneau <bchesneau@gmail.com> =
wrote:
>>=20
>>> Looking at the code of mem3 it seems that all databases are =
oblgatory
>>> fragmented (sharded). I understand that you can have only 1 shard /
>>> database but then it will be still stored at
>>> `<database_dir>/shard/<dbname>`.
>>>=20
>>> Is there any plan of having non fragmented databases stored in their
>>> own place ie `<database_dir>/<dbname>`  ? Would be interesting when
>>> you want to manage different policies of backup depending on the =
type
>>> of database (fragmented or not). Which place should I look to make =
it
>>> happen?
>>>=20
>>> - benoit
>>=20
>> I'm not entirely certain I understand what you're looking for there =
Benoit.  I classify databases in two categories: "clustered" and =
"local".
>=20
>>=20
>> A clustered database is available via the clustered HTTP interface =
and has its data stored in files inside =
<database_dir>/shards/<range>/<dbname>.  This is true even for a Q=3D1,N=3D=
1 database where this is only one file for the database in the entire =
cluster.
>>=20
>> On the other hand, a local database exists on a specific node in the =
cluster, and is not accessible via the clustered HTTP interface.  The =
data for this database is stored at <database_dir>/<dbname>.
>>=20
>> What exactly did you mean by a non-fragmented database?  It sort of =
sounds like you're wanting to conflate a Q=3D1,N=3D1 clustered database =
with a local database, but those are very different concepts (not least =
because multiple nodes in a cluster can have a local database with the =
same name but different content).  I can honestly say we've never =
considered storing the files for Q=3D1 or Q=3D1,N=3D1 databases =
somewhere different than other clustered databases.
>=20
> A fragmented or sharded database is the clustered one. I call them
> fragmented databases since afaik a clustered database is spitted in
> multiple pieces.
>=20
>>=20
>> Can you say more about your motivations here?  You talked about =
backup policies; while I could absolutely see wanting to implement a =
different backup policy for N=3D1 databases it's not obvious to me why =
you'd want to treat Q=3D1 and Q>1 databases separately.
>=20
> Having them in there own folder instead of shard them them easily more
> distinct on the fs than having them in shards/ and i could easily
> find db.couch by its name without using the API on the disk.
>=20
> Also I am wondering right now how the transition from a the current
> couchdb to the new one can be done. It would be quite easier imo to
> have 1 HTTP layer and flag the  "special databases" to not expose them
> in the HTTP API except for the admins.
>=20
> - benoit