incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Joseph Davis (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (COUCHDB-1270) Rewrite the view engine
Date Mon, 05 Sep 2011 19:16:09 GMT

     [ https://issues.apache.org/jira/browse/COUCHDB-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Paul Joseph Davis updated COUCHDB-1270:
---------------------------------------

    Attachment: 0004-Remove-the-old-view-engine.patch
                0003-Create-the-couch_mrview-application.patch
                0002-Create-the-couch_index-application.patch
                0001-Minor-changes-for-new-indexing-engine.patch

@Filipe

Good catch on both. For point one, I just removed the nil default. In my refactoring I undid
almost all of these setting record fields to nil because its fairly silly to use a nonstandard
value for undefined when the standard value is literally, undefined. One place I didn't change
this yet was for the qserver variable because of how couch_query_servers:stop_doc_map is written
and I was wanting to minimize the scope of these changes.

As to 2, I added the file removal in the cancellation function. One thing this doesn't address
is a generalized method for handling the new compaction lifetime model. We'll have to revisit
this a bit to try and get a better abstraction for compaction.

The timing tests I was running were against the views in [1] which you've used before for
these types of tests. The test is just a simple "time curl http://127.0.0.1:5984/indexer_test/_design/test/_view/view1?limit=0"
or w/e the URL is for that design doc. Fairly consistently the new-views are a minute or two
faster than trunk. So, nothing super improved but convincing enough to me that I haven't trashed
performance.

Also, I did spend a couple days profiling the view engine and comparing a couple dirty hacks.
I'm convinced we can shorten these times even more with some more refactoring and playing
with some other optimizations but they were leading me deep into the couch_query_servers and
couch_os_process. I plan on backporting some refactoring for these parts next and that should
set us up for some of the more advanced changes to the view engine.

Trunk: 26m29.809s
New-Views: 24m50.923s

[1] http://fdmanana.couchone.com/indexer_test

@Benoit

I'd agree on turnign couch_index_api into a behavior but I decided to delay that until after
we update the various applications and build system so that we have a proper ERL_LIBS directory.
For behaviours to work they need to be importable and that's not the easiest given our current
structure. Though it'll definitely be something to add in the future.

For supervisors I think that could be a good idea, but I'm not the most familiar with them
and their side effects to know if they'd be a good fit here. You mentioned on IRC that you'd
take a swing at implementing that and I'd be quite interested to see what you come up witih.

> Rewrite the view engine
> -----------------------
>
>                 Key: COUCHDB-1270
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1270
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: JavaScript View Server
>            Reporter: Paul Joseph Davis
>         Attachments: 0001-Minor-changes-for-new-indexing-engine.patch, 0002-Create-the-couch_index-application.patch,
0003-Create-the-couch_mrview-application.patch, 0004-Remove-the-old-view-engine.patch
>
>
> The view engine has been creaky and cluttered. As shown by GeoCouch, adding new indexers
basically involves copying the entire view engine and hacking the parts that are different.
In short, the opposite of good engineering.
> Over the last couple weeks I've refactored the view engine and reimplemented the map/reduce
view engine. These changes are 100% internal and no external behavior has changed. Performance
is just a tiny bit better than trunk. I did do some playing trying to improve view update
times and there are some dances we could do, but for the time being I wanted to keep the same
general architecture for updates so that the changes are minimal.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message