couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <rnew...@apache.org>
Subject Re: Ad-hoc filtering
Date Tue, 26 Feb 2013 15:43:51 GMT
List functions are no substitute for views. You're right that they
exist to support output formats other than JSON, and you can abuse
them a little to filter (a little). Do this inappropriately, though,
and you're back to a table scan worst-case result.

If your needs are truly ad-hoc, and you won't add an ad-hoc querying
system like Lucene, then I've run out of things to suggest.

B.

On 26 February 2013 15:18, Guillaume Buisson
<guillaumeb@doremitechno.org> wrote:
> Docs for lists are available here:
> http://wiki.apache.org/couchdb/Formatting_with_Show_and_List
>
> I used this method to do basic "on the fly" filters and pagination,
> The list acts as some kind of decorator for view, my feeling is that it's
> been implemented to do format conversions, but you can use it for other
> things...
> the req var contains all request arguments (headers, search etc.), it is
> then easy to test each rows/keys and only return the corresponding docs.
>
> This will however come at a cost, the way i see it, all data from a view is
> passed to the list and evaluated on request with no "caching strategy",
> if your view fetches thousands of docs, this can be very ressource
> hungry...
>
>
> 2013/2/26 Guillaume Buisson <guillaumeb@doremitechno.org>
>
>> Why don't you use couchDB lists ?
>>
>> On lists you can fetch search parameters and filter data accordingly,
>> i know this is not optimal performance wise, but still it's better than
>> pushing views each time.
>>
>>
>> 2013/2/26 Jan Lehnardt <jan@apache.org>
>>
>>>
>>> On Feb 26, 2013, at 14:54 , Troy Farrell <troy@entheossoft.com> wrote:
>>>
>>> > It's probably not a popular opinion, but this application does not sound
>>> > well-suited for CouchDB.  If you are creating views all the time, using
>>> > them once and still downloading all the data for post-processing, then
>>> > something is clearly wrong.  We may be able to help you restructure the
>>> > data and views so you need fewer of them, but if you truly need lots of
>>> > dynamic queries, CouchDB may not be the best choice for your
>>> application.
>>> > (I would love for someone on this list to tell me I'm wrong.)
>>>
>>> I think determining when CouchDB is *not* a good fit is crucial. I for one
>>> am not interested in unhappy users.
>>>
>>> Best
>>> Jan
>>> --
>>>
>>>
>>> >
>>> > Troy
>>> >
>>> > On Mon, Feb 25, 2013 at 5:50 PM, Daniel Gonzalez <gonvaled@gonvaled.com
>>> >wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> It is not possible for me to define and manage all possible views that
>>> my
>>> >> application needs. Each time that I need to access data with a new
>>> >> filtering mechanism I have to:
>>> >>
>>> >>   1. Check if that view, or a similar one, already exists. (How would
I
>>> >>   call this if I had created this half a year ago?)
>>> >>   2. If it doesn't, I must create it (javascript, not my piece of
>>> cake. I
>>> >>   am most of the time in python land).
>>> >>   3. I have to push the new design doc to all my servers (did you also
>>> >>   push to that-server-that-you-never-use? Do you really need to push
>>> this
>>> >>   view to that server? I don't remember, really.)
>>> >>   4. I have to trigger the new view (which can take some time in
>>> databases
>>> >>   with lots of docs), in all servers.
>>> >>
>>> >> So now you need to filter for False instead of True? Great! Rinse and
>>> >> repeat! At the end of the day I end up with dozens of views, some of
>>> them
>>> >> outdated, which I am not using anymore, but I am unable to maintain.
>>> >>
>>> >> Besides, defining lots of views becomes impractical: the more views
you
>>> >> have in a design doc, the longer it takes a refresh when the design
>>> doc is
>>> >> modified. And the more views, the more space the database uses.
>>> >>
>>> >> At some point this becomes very painful. Which means that for some
>>> data I
>>> >> just do client-filtering.
>>> >>
>>> >> Very often I must perform the following operations:
>>> >>
>>> >>   1. Access a view
>>> >>   2. Filter the documents in the view according to some criteria
>>> (which is
>>> >>   not supported by a view)
>>> >>   3. Paginate the resulting documents
>>> >>
>>> >> Since the filtering is done in the client, I am forced to get all
>>> >> documents. I will discard most of them, but since I do not know which
>>> ones,
>>> >> I still must get them all. This is very slow.
>>> >>
>>> >> Since for some data I do view-filtering, and for other data I do
>>> >> client-filtering, some operations are taking a very long time to
>>> complete,
>>> >> whereas others are completed very fast. Which is confusing for the
>>> user.
>>> >>
>>> >> It would be great if couchdb supported field filtering. For example
I
>>> would
>>> >> pass a dictionary like this: { 'available' : true, 'user' : 'user1'
}
>>> And I
>>> >> would get only the documents which satisfy the filter. Using a single
>>> view.
>>> >> A simple "is equal to" operation for the given fields would go a
>>> looong way
>>> >> to solve most of my filtering needs.
>>> >>
>>> >> The view engine would perform the standard indexing, and the filtering
>>> >> would be an extra layer (slowing things down a bit, probably). It
>>> would be
>>> >> a compromise between view indexing and client filtering.
>>> >>
>>> >> Does this mechanism already exist? Does it make sense? Would it ever
be
>>> >> implemented?
>>> >>
>>> >> Thanks
>>> >> Daniel Gonzalez
>>> >>
>>>
>>>
>>
>
> This email has been verified by Google Postini filtering system

Mime
View raw message