Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of dionne@dionne-associates.com
 designates 69.89.21.11 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default;
 d=dionne-associates.com;
	h=Received:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Content-Transfer-Encoding:Message-Id:References:To:X-Mailer:X-Identified-User;
	b=RRr368Xu0RH4YWl8SHWzKzfUv7nbKVZea4eNNVU8evmWPaDqoMUdYC1pvkX6+7p5I+sFWDwu53NVTM6g5HjMvjIkl+mE5v7aNeVFSmA32TP1odoFitD28uGoEi6u70gY;
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1077)
Subject: Re: two view questions: group=true, inverted indices
From: Robert Dionne <dionne@dionne-associates.com>
In-Reply-To: <e2111bbb1002071529t6aaa8e23p908ef7af72d0075e@mail.gmail.com>
Date: Mon, 8 Feb 2010 06:31:04 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <D8851536-4606-49EB-B3FA-CC674E1686BD@dionne-associates.com>
References: <fc9cb8941002071515x1a5cae8eha8fe569068ac3232@mail.gmail.com>
 <e2111bbb1002071529t6aaa8e23p908ef7af72d0075e@mail.gmail.com>
To: user@couchdb.apache.org


On Feb 7, 2010, at 6:29 PM, Paul Davis wrote:

> On Sun, Feb 7, 2010 at 6:15 PM, Harold Cooper <harold@mit.edu> wrote:
>> Hi there,
>>=20
>> I'm new to CouchDB and have two questions about the use of mapreduce
>> in views.
>>=20
>> 1.
>> As far as I can tell, even when I pass group=3Dtrue to a view,
>> reduce(keys, values) is still passed different keys,
>> e.g. keys =3D [["a", "551a50e574ccd439af28428db2401ab4"],
>> ["b", "94d13f9e969786c6d653555a7e94f61e"]].
>>=20
>=20
> Even when you query with group=3Dtrue, the ungrouped reduction is =
still
> calculated. Generally you should be able to just ignore such things.
>=20
>> Isn't the whole point of group=3Dtrue that this shouldn't happen?
>>=20
>>=20
>> 2.
>> When I've read about mapreduce before, a classic example use is
>> constructing an inverted index. But if I make a view like:
>> {
>> map: "function(doc) {
>>  var words =3D doc.text.split(' ');
>>  for (var i in words) {
>>    emit(words[i], [doc._id]);
>>  }
>> }",
>> reduce: "function(keys, values) {
>>  // concatenate the lists of docIds together:
>>  return Array.prototype.concat.apply([], values);
>> }"
>> }
>> then couchdb complains that the reduce result is growing too fast.
>>=20
>> I did read that this is the way things are, but it's too bad because
>> it would be a perfect application of mapreduce, and the only other
>> text search option I've heard of is couchdb-lucene which doesn't
>> sound nearly as fun/elegant.
>>=20
>> Is there another way to approach this?
>> Should I just not reduce and end up with one row per word-occurrence?
>=20
> CouchDB Map/Reduce isn't like Google Map/Reduce. Its much more like
> the old school map/reduce pattern that expects to be calculating a
> single reduction value. The CouchDB internals make doing things like
> inverted indices hard. The 'proper' way would be to do as you say and
> return a single row per key with only some intermediary values handled
> by reductions.
>=20
> Also, while couchdb-lucene may not present near as much fun, its got
> quite a bit to it. Full-Text indexing is hard. Many examples show it
> as nothing more than an inverted index, but that's hiding 95% of the
> knowledge on information retrieval and scoring algorithms that are in
> Lucene. And there's the integration with Tika to do things like
> attachment indexing. I quite dislike Java but I've come to accept that
> there really isn't much competition that's compatible with CouchDB's
> document model.
>=20

I think it does have challenges and couchdb-lucene offers a good =
solution for most use cases, plus it's mature and well known, but
at some point, perhaps post 1.0 I think a native FTI implementation will =
add a lot of value to CouchDB if only by removing the dependency
on Java.=20


> HTH,
> Paul Davis
>=20
>> Thanks for any help,
>> and sorry if this has been covered before, I did try to search around =
first.
>> --
>> Harold
>>=20