couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Gonzalez <gonva...@gonvaled.com>
Subject Re: Ad-hoc filtering
Date Tue, 26 Feb 2013 20:19:36 GMT
I see. I was just hoping that some of that ad-hoc querying was already part
of core couchdb.

Just curious: is there a major technical reason not to have that capability
integrated in couchdb, or is just a matter of separation of
responsibilities and specialization? I realize that there are mature
projects out there which have very powerful querying capabilities, but for
simple setups a limited querying support would be very useful.

On Tue, Feb 26, 2013 at 4:43 PM, Robert Newson <rnewson@apache.org> wrote:

> List functions are no substitute for views. You're right that they
> exist to support output formats other than JSON, and you can abuse
> them a little to filter (a little). Do this inappropriately, though,
> and you're back to a table scan worst-case result.
>
> If your needs are truly ad-hoc, and you won't add an ad-hoc querying
> system like Lucene, then I've run out of things to suggest.
>
> B.
>
> On 26 February 2013 15:18, Guillaume Buisson
> <guillaumeb@doremitechno.org> wrote:
> > Docs for lists are available here:
> > http://wiki.apache.org/couchdb/Formatting_with_Show_and_List
> >
> > I used this method to do basic "on the fly" filters and pagination,
> > The list acts as some kind of decorator for view, my feeling is that it's
> > been implemented to do format conversions, but you can use it for other
> > things...
> > the req var contains all request arguments (headers, search etc.), it is
> > then easy to test each rows/keys and only return the corresponding docs.
> >
> > This will however come at a cost, the way i see it, all data from a view
> is
> > passed to the list and evaluated on request with no "caching strategy",
> > if your view fetches thousands of docs, this can be very ressource
> > hungry...
> >
> >
> > 2013/2/26 Guillaume Buisson <guillaumeb@doremitechno.org>
> >
> >> Why don't you use couchDB lists ?
> >>
> >> On lists you can fetch search parameters and filter data accordingly,
> >> i know this is not optimal performance wise, but still it's better than
> >> pushing views each time.
> >>
> >>
> >> 2013/2/26 Jan Lehnardt <jan@apache.org>
> >>
> >>>
> >>> On Feb 26, 2013, at 14:54 , Troy Farrell <troy@entheossoft.com> wrote:
> >>>
> >>> > It's probably not a popular opinion, but this application does not
> sound
> >>> > well-suited for CouchDB.  If you are creating views all the time,
> using
> >>> > them once and still downloading all the data for post-processing,
> then
> >>> > something is clearly wrong.  We may be able to help you restructure
> the
> >>> > data and views so you need fewer of them, but if you truly need lots
> of
> >>> > dynamic queries, CouchDB may not be the best choice for your
> >>> application.
> >>> > (I would love for someone on this list to tell me I'm wrong.)
> >>>
> >>> I think determining when CouchDB is *not* a good fit is crucial. I for
> one
> >>> am not interested in unhappy users.
> >>>
> >>> Best
> >>> Jan
> >>> --
> >>>
> >>>
> >>> >
> >>> > Troy
> >>> >
> >>> > On Mon, Feb 25, 2013 at 5:50 PM, Daniel Gonzalez <
> gonvaled@gonvaled.com
> >>> >wrote:
> >>> >
> >>> >> Hi,
> >>> >>
> >>> >> It is not possible for me to define and manage all possible views
> that
> >>> my
> >>> >> application needs. Each time that I need to access data with a
new
> >>> >> filtering mechanism I have to:
> >>> >>
> >>> >>   1. Check if that view, or a similar one, already exists. (How
> would I
> >>> >>   call this if I had created this half a year ago?)
> >>> >>   2. If it doesn't, I must create it (javascript, not my piece
of
> >>> cake. I
> >>> >>   am most of the time in python land).
> >>> >>   3. I have to push the new design doc to all my servers (did you
> also
> >>> >>   push to that-server-that-you-never-use? Do you really need to
push
> >>> this
> >>> >>   view to that server? I don't remember, really.)
> >>> >>   4. I have to trigger the new view (which can take some time in
> >>> databases
> >>> >>   with lots of docs), in all servers.
> >>> >>
> >>> >> So now you need to filter for False instead of True? Great! Rinse
> and
> >>> >> repeat! At the end of the day I end up with dozens of views, some
of
> >>> them
> >>> >> outdated, which I am not using anymore, but I am unable to maintain.
> >>> >>
> >>> >> Besides, defining lots of views becomes impractical: the more views
> you
> >>> >> have in a design doc, the longer it takes a refresh when the design
> >>> doc is
> >>> >> modified. And the more views, the more space the database uses.
> >>> >>
> >>> >> At some point this becomes very painful. Which means that for some
> >>> data I
> >>> >> just do client-filtering.
> >>> >>
> >>> >> Very often I must perform the following operations:
> >>> >>
> >>> >>   1. Access a view
> >>> >>   2. Filter the documents in the view according to some criteria
> >>> (which is
> >>> >>   not supported by a view)
> >>> >>   3. Paginate the resulting documents
> >>> >>
> >>> >> Since the filtering is done in the client, I am forced to get all
> >>> >> documents. I will discard most of them, but since I do not know
> which
> >>> ones,
> >>> >> I still must get them all. This is very slow.
> >>> >>
> >>> >> Since for some data I do view-filtering, and for other data I do
> >>> >> client-filtering, some operations are taking a very long time to
> >>> complete,
> >>> >> whereas others are completed very fast. Which is confusing for
the
> >>> user.
> >>> >>
> >>> >> It would be great if couchdb supported field filtering. For example
> I
> >>> would
> >>> >> pass a dictionary like this: { 'available' : true, 'user' : 'user1'
> }
> >>> And I
> >>> >> would get only the documents which satisfy the filter. Using a
> single
> >>> view.
> >>> >> A simple "is equal to" operation for the given fields would go
a
> >>> looong way
> >>> >> to solve most of my filtering needs.
> >>> >>
> >>> >> The view engine would perform the standard indexing, and the
> filtering
> >>> >> would be an extra layer (slowing things down a bit, probably).
It
> >>> would be
> >>> >> a compromise between view indexing and client filtering.
> >>> >>
> >>> >> Does this mechanism already exist? Does it make sense? Would it
> ever be
> >>> >> implemented?
> >>> >>
> >>> >> Thanks
> >>> >> Daniel Gonzalez
> >>> >>
> >>>
> >>>
> >>
> >
> > This email has been verified by Google Postini filtering system
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message