incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Barker <norman.bar...@gmail.com>
Subject Re: multiview on github
Date Mon, 23 Aug 2010 03:22:32 GMT
Bob,

I am testing on 10000+ documents, I appreciate that we need to
establish when a multi-process as opposed to a tbd (suggestions
welcome) approach is required. The startkey / endkey is an issue
though, is there a better way to test inclusion?

The speed of the multiview is directly linked to the size of the
smallest view result though, so total documents isn't a factor.

I am still thinking about fti, I am testing with clucene, but the
external handler problem is the same, how to make it stream in order.

I will fix the local_dev.ini problem tomorrow, couchdb should be in
version control.

Any hints on how to test inclusion are appreciated, it will greatly
speed up collation.

thanks,

Norman



On Sun, Aug 22, 2010 at 4:15 PM, Robert Newson <robert.newson@gmail.com> wrote:
> I'm concerned about the performance of this on non-trivial databases,
> given the iteration of all items between startkey and endkey. I don't
> have time to test it this week but I'd be interested to hear the time
> it took to do a multiview on two views of, say, a million rows each
> (especially as compared to the two normal view calls).
>
> I was also intrigued to see the code handles fti too, a problem I have
> spent some time thinking about without finding a satisfactorily
> performant solution too. I note that, as written, it doesn't appear to
> work because the fti call (I'm assuming couchdb-lucene) will only
> return the top N matching hits, so at best you can filter those
> against another view (perhaps that's useful?). The trick to merging a
> view and an fti result together would be to get the results from both
> in the same order and step through the rows, filtering as you go.
> Sorting in Lucene has a large memory hit so I gave up on that
> solution.
>
> Finally, your patch appears to add two generated files (local_dev.ini
> and etc/init.d/couchdb) to the branch which should be fixed (add your
> settings to default.init.tpl.in instead).
>
> I should end by saying that if the problems above can be solved then
> this would be a very useful addition to CouchDB and one that is
> frequently requested. It might also be a model for multi-machine
> views.
>
> B.
>
> On Sun, Aug 22, 2010 at 10:45 PM, Norman Barker <norman.barker@gmail.com> wrote:
>> I would like to take this multiview code and have it added to trunk if
>> possible, what are the next steps?
>>
>> thanks,
>>
>> Norman
>>
>> On Wed, Aug 18, 2010 at 11:44 AM, Norman Barker <norman.barker@gmail.com> wrote:
>>> I have made
>>>
>>> http://github.com/normanb/couchdb
>>>
>>> which is a fork of the latest couchdb trunk with the multiview code
>>> and tests added.
>>>
>>> If geocouch is available then it can still be used.
>>>
>>> There are a couple of questions about the multiview on the user /dev
>>> list so I will be adding some more test cases during today.
>>>
>>> thanks,
>>>
>>> Norman
>>>
>>> On Tue, Aug 17, 2010 at 9:23 PM, Norman Barker <norman.barker@gmail.com>
wrote:
>>>> this is possible, I forked geocouch since I use it, but I have already
>>>> separated the geocouch dependencies from the trunk.
>>>>
>>>> I can do this tomorrow, certainly be interested in any feedback.
>>>>
>>>> thanks,
>>>>
>>>> Norman
>>>>
>>>>
>>>>
>>>> On Tue, Aug 17, 2010 at 7:49 PM, Volker Mische <volker.mische@gmail.com>
wrote:
>>>>> On 08/18/2010 03:26 AM, J Chris Anderson wrote:
>>>>>>
>>>>>> On Aug 16, 2010, at 4:38 PM, Norman Barker wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have made the changes as recommended, adding a test case
>>>>>>> multiview.js and also adding the userCtx to open the db.
>>>>>>>
>>>>>>> I have also forked geocouch and this is available here
>>>>>>>
>>>>>>
>>>>>> this patch seems important (especially as people are already asking
for
>>>>>> help using it on user@)
>>>>>>
>>>>>> to get it committed, it either must remove the dependency on GeoCouch,
or
>>>>>> become part of CouchDB when (and if) GeoCouch becomes part of CouchDB.
>>>>>>
>>>>>> Is it possible / useful to make a version that doesn't use GeoCouch?
And
>>>>>> then to make the GeoCouch capabilities part GeoCouch for now?
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>
>>>>> Hi Norman,
>>>>>
>>>>> if the patch is ready for trunk, I'd be happy to move the GeoCouch bits
to
>>>>> GeoCouch itself (as GeoCouch isn't ready for trunk yet).
>>>>>
>>>>> Lately I haven't been that responsive when it comes to GeoCouch, but
that
>>>>> will change (in about a month) after holidays and FOSS4G.
>>>>>
>>>>> Cheers,
>>>>>  Volker
>>>>>
>>>>
>>>
>>
>

Mime
View raw message