Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: couchdb-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of jchris@gmail.com designates
 72.14.220.159 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=message-id:date:from:sender:to:subject:mime-version:content-type:content-transfer-encoding:content-disposition:x-google-sender-auth;
        b=AZk3/nbiNay2H9mV8HPNN63TSbQMAxyk5WEoEt7KcbAMDXYIHHi9OG7ZyKsIY3iYZRfy4RLU8ZH3TYZwzIJDA7iyOdVJ0cPcCrShEmPVu6b4qK+46Q1b+FMbR9HQvpzrjHenPZhJ1zmKOmv23iD5p40Bh+0qmVVHZz20auwW1LY=
Message-ID: <e282921e0803211455s2ae067fet5aecc94b0561b144@mail.gmail.com>
Date: Fri, 21 Mar 2008 14:55:26 -0700
From: "Chris Anderson" <jchris@mfdz.com>
Sender: jchris@gmail.com
To: couchdb-user@incubator.apache.org
Subject: Sphinx integration (was: Working on Lucene)
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

On Fri, Mar 21, 2008 at 1:34 PM, Jan Lehnardt <jan@apache.org> wrote:
>  Thanks for the input. This is actually an implementation detail of
>  the Indexer, but I agree that this should be supported. I also think
>  we should have some standard way here so other search solutions
>  can be plugged in without breaking things.
>

Jan,

Some thoughts about Sphinx integration.

The HTTP API as it currently stands (just the ability to page through
an entire view) is sufficient to implement Sphinx indexing on views as
an external process.

However, Sphinx has the requirement that the documents it indexes each
have a unique, numerical id. Using the CouchDB document ID would not
be advised in that case. Using a map function the emits once per
document (or using Reduce/Combine when it becomes available) coupled
with a function to deterministically convert CouchDB document ids into
integers should make for views which can be easily indexed by Sphinx.

The map function might look like this

function(doc) {
  if (doc.title) {
    map(docIDtoInteger(doc.id), doc.title);
  }
}

It's too bad that Sphinx doesn't support arbitrary strings as document
IDs, but I'm sure there are plenty of reversible string-to-integer
mappings that could be used. In that case Sphinx would be queried and
return a list of matching integers IDs, which could be mapped back to
CouchDB document IDs, and then retrieved from the Couch.

This thought experiment is encouraging because it shows that even
without integration into CouchDB, some very useful custom full-text
indexes could be created. AFAIK Sphinx's support for updating indexes
is limited to merging new documents into the index, so it would have
little use for an API to find view-rows which have been changed or
removed. Luckily, index rebuild is lightning fast.


-- 
Chris Anderson
http://jchris.mfdz.com