couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: couch_gen_btree: pluggable storage / tree engines
Date Sun, 01 Feb 2009 03:56:39 GMT
Martin,

Very cool ideas. We've been discussing erlang plugins. The
conversation has generally gotten as far as, "erlang plugins... yeah
we should have those."

As you've noticed, the couch sources depend fairly non-transparently
on the couch_btree implementation. Here's my take on the situation:

1. Erlang plugins are a Good Idea &trade;
2. We should probably focus on at least two types of plugins: storage
and indexing.
3. MySQL is repulsive.

1. I'm going to assume consensus.
2. I'm pretty sure that the two types of plugins are quite different
and will even require different semantics in the _design docs all the
way down to calls to emit.
3. Well, I'm going to assume consensus here too.

Anyway, I think the general steps forward are probably to figure out
what exactly we want to abstract, how to abstract it, and then start
abstracting. Abstractions FTW!

In terms of storage, I'm not entirely certain on the specifics.
There's quite of the API in the raw database layer that i haven't
dealt with yet. It'd be super fun to have specific file systems for
things like multiple servers on a single rack vs the distributed
CouchDB for distant servers etc.

In terms of view indexing, my mind is already racing on the
possibilities of where to push things around to make indexes
selectable in the _design docs etc. This could be really be a fun
project.

Keep up with the reading and thinking. This is something I look
forward to seeing implemented.

HTH,
Paul Davis


On Sat, Jan 31, 2009 at 9:49 AM, Martin Scholl <ms@diskware.net> wrote:
> Hello all,
>
>
> although I am still doing a CouchDB review to better understand its
> design, I like to ask for comments for a tiny idea.
> I would like to add another index structure to CouchDB (a Merkle-Tree)
> and come up with asking myself what the best way of doing this would be.
> I have a rough guess of how closely couch_btree is knit into CouchDB.
> Therefore I would like to hear from you experienced developers comments
> on some of my ideas:
>
> My suggestion is a MySQL-ly approach (pluggable engines) for CouchDB,
> that is to factor out several components into generic behaviours:
> e.g.
> - a couch_gen_tree:
>        abstracts access to couch_btree
>
> Maybe even a
> - couch_gen_storage
>        e.g. file system, file storage access, etc.
>
> - couch_gen_replicator
>        an imperative approach to tree / storage replication.
>
> As I said: I am new to CouchDB's code so I cannot really estimate how
> the current layering approach looks like, and whether we can even split
> out the 3 components.
>
> Imho there would be several benefits in having this flexibility brough
> by couch_gen_*:
> - new use-cases for CouchDB:
>        - R-trees: for adding another way of querrying
>          documents (e.g. nearest neighbour search)
>        - genome databases
>        - a special datastructure for indexing tags
>        - ...
> - with a flexible storage layer, CouchDB could ran on top of other
> infrastructures and products: like S3, SimpleDB, AppEngine, etc.
> - (the following is just a guess:) a cleaner CouchDB codebase with a
> clear layering and separation of components
> - possibly (again, just a guess): with the plugin approach, we can more
> easily support advanced indexing and db management schemes, like
> distributed storage access, distributed transactions, etc.
>
>
> Martin
>

Mime
View raw message