incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brad King" <brk...@gmail.com>
Subject Re: Iterating over/paging through large sets of documents
Date Mon, 28 Jul 2008 03:24:43 GMT
I think the iterator mechanisms in couchdb are one of the things that
are underestimated. I've been prototyping an inventory system for a
few weeks, and couchdb views are soo much easier to paginate than,
say, a SQL server DB, because the indexes are explicit, and exposed
directly as views. What I do is simply request a key and a count of +1
more than I need for a page, then use that last value as my start key
for the next page. Simple, clean, fast. I don't even bother trying to
use and end key. Its not feasible for most applications of paging.

On Fri, Jul 25, 2008 at 7:13 PM, Chris Anderson <jchris@grabb.it> wrote:
> John,
>
> I wrote a simple algorithm to page through all the results of a view,
> yielding them one key (with all it's values) at a time. I also have a
> simple loop for iterating through all_docs. You can see both of them
> here:
>
> http://github.com/jchris/couchrest/tree/master/lib/pager.rb
>
> I'm sure I could generalize the all_docs pager to work for any view,
> but for now it works for me.
>
> Using "count" is fine when you want to limit a set - it's just using
> "skip" that is discouraged.
>
> Chris
>
> On Fri, Jul 25, 2008 at 1:43 PM, John Evans <john@jpevans.com> wrote:
>> I think I read somewhere on the wiki that it's better to use startkey/endkey
>> to grab "pages" of documents, as opposed to using "count" because "count"
>> still has to materialize the entire set of results before filtering to the
>> specified count (or something like that).
>>
>> In a case where you have a reasonable set of start/end key pairs in advance,
>> this makes perfect sense, but what is the best practice for paging through a
>> set of documents where you don't know any keys?  For example, can can you
>> start with count=10 to get the first ten results and then set the startkey
>> equal to the key of the 10th result from the first request and use count=11
>> to get the next 10 documents (ignoring the first one that overlaps with the
>> last set) or will this run into the same problem?
>>
>> It feels like I am making this more complicated than it needs to be...
>> please tell me this is the case :)
>>
>> Thanks,
>> -
>> John
>>
>
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>

Mime
View raw message