couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Branca <chewbra...@apache.org>
Subject On Alternative View Engines
Date Sun, 30 Jun 2013 21:37:32 GMT
The discussion of alternative approaches to view engines is one that
bubbles up semi regularly, with the latest addition for a Lua native query
server described in COUCHDB-1842 by Alexander. Lua is a great language for
embedding into systems and provides powerful sandboxing facilities.

I'm very intrigued by optimizing for the standard use case, where a user
wants to build a simple secondary index on their data, and then use a built
in reduce function. I think we can find a solution that allows a user to
define a doc level transformation in a DSL or query language or some other
approach that allows us to keep the view generation functionality within
the Erlang VM and avoid the overhead costs of using an external engine.

I do think it makes sense to have an external engine for flexibility, and
allowing us to focus on the simple cases while providing a fallback for
more complex user defined functions.

To experiment with different approaches, I built a Lisp interpreter on top
of Erlang with the premise of white listing the entire language, allowing
explicit control over what the user can and cannot do in view functions.
You can see it here: [Lispenport](https://github.com/chewbranca/lispenport).
It's by no means a full solution, but it has some interesting properties
such as really just being syntactic sugar on top of Erlang and all
constructs are direct Erlang terms, even lambdas are just Erlang funs.

Now, while I would be intrigued by a Lisp DSL for user defined functions in
CouchDB, I didn't expect that to be well received by everyone, so I've
considered this just an experiment. If we were going to take this approach,
I would rather take Lisp Flavored Erlang (LFE, another project by Robert
Virding along with Luerl, and also erlog, a Prolog interpreter in Erlang),
and rip out all the pieces we would not want a user to access and use LFE
as a base starting point. LFE compiles down to intermediate Erlang bytecode
and is designed to follow Erlang functionality, making it a nice option for
building a view engine to execute in the Erlang VM.

I've also toyed around with the idea of building a NIF around [JQ](
http://stedolan.github.io/jq/) which is a great application for slicing and
dicing json data structures written in C.

So my general proposal for discussion is that we build a minimal DSL of
some sort, providing fast and simple doc manipulations that executes
securely in the Erlang VM, and then we abstract out all functionality for a
"full" view engine, list functions, show functions, etc to a separate
engine that is easily swappable and not required for standard functionality.

Thoughts?


-Russell

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message