couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Klaus Trainer <>
Subject Re: Summary of IRC meeting in #couchdb-meeting, Wed Jan 30 20:01:17 2013
Date Wed, 30 Jan 2013 23:14:33 GMT
I couldn't attend the meeting. But still having such a summary is
awesome :)


On Wed, 2013-01-30 at 21:39 +0000, ASF IRC Services wrote:
> Members present: Noah, kocolosk, Jarda, jan____, wendall911, benoitc, drsm79, daleharvey_,
rnewson, dch, chewbranca, Kxepal
> ----------------
> Meeting summary:
> ----------------
> 1. Preface
>   a. last weeks** minutes here
(dch, 1)
> 2. 1.3.x blockers
>   b. is useful btw. Only 1654 remains. (dch,
> 3. rcouch merge
>   y. benoitc working on introducting rebar to couch, as part of rcouch merge (dch, 3)
> 4. fauxon, bring me all your UX goodness.
>   >. ASF CLAs underway for fauxtoneers (dch, 4)
>   >. jan to add fauxton component to JIRA (jan____, 4)
> 5. 1.3.x CORS
> 6. couchdbconf was awesome
>   d. jan to mail about CouchDB Conf (jan____, 6)
> 7. benchmarking
>   a. (dch, 7)
>   j. (dch, 7)
>   r. dch to find some people with large docs/complex views for testing (dch, 7)
>   >. kxepal to contribute real world docs and views for testing - huge thanks! (dch,
> 8. packages
>   a. wendall911 to come back to ML about EPEL packages (dch, 8)
> --------
> Actions:
> --------
> - benoitc working on introducting rebar to couch, as part of rcouch merge (dch, 20:36:09)
> - ASF CLAs underway for fauxtoneers (dch, 20:43:49)
> - jan to add fauxton component to JIRA (jan____, 20:51:04)
> - jan to mail about CouchDB Conf (jan____, 20:55:34)
> - dch to find some people with large docs/complex views for testing (dch, 21:04:23)
> - kxepal to contribute real world docs and views for testing - huge thanks! (dch, 21:18:26)
> - wendall911 to come back to ML about EPEL packages (dch, 21:21:22)
> IRC log follows:
> # 1. Preface #
> 20:02:17 [dch]: #info last weeks** minutes here
> 20:03:39 [dch]: so topics our usual 1.3.x where are we at, I'm keen to hear what went
on at couchhack after I had to return, items from last meeting etc.
> 20:03:54 [dch]: hey Adam welcome :-) 
> 20:04:02 [kocolosk]: hiya
> 20:04:17 [Noah]: dch: we should also move around each member in the channel to ask if
they have any business to discuss
> 20:04:24 [rnewson]: the view sig thing is a blocker.
> 20:04:26 [Noah]: dch: or some variation of that
> 20:04:32 [daleharvey_]: aloha
> 20:04:54 [dch]: ok, let's do that then
> 20:05:02 [dch]: Filipe welcome too :-)
> 20:05:02 [dch]: and dale
> 20:05:39 [drsm79]: hello meeting room
> 20:05:49 [dch]: welcome Simon, I have a face for the name now .
> 20:05:56 [drsm79]: :0
> 20:06:02 [drsm79]: :)
> 20:06:03 [drsm79]: a beard for the name, too
> 20:06:09 [dch]: OK, quick round check for items, then lets get cracking.
> 20:06:17 [dch]: fdmanana:  Jarda  JasonSmith any other topics?
> 20:06:39 [dch]: kocolosk: Noah rnewson ? Other than a fine meal chez Newson atm.
> 20:06:50 [dch]: benoitc: chewbranca daleharvey_ anything?
> 20:07:02 [Noah]: dch: i am thinking it might be easier to just go round each person -
topics will come up naturally
> 20:07:02 [Jarda]: I'm actually intersted in the upcoming Futon and if it needs some extra
hands? (If this is a suitable forum for that)
> 20:07:09 [Noah]: dch: collecting topics at the start just seems to slow us down
> 20:07:17 [benoitc]: sorry
> 20:07:17 [dch]: Noah: you need to decide if you're running the meeting or not.
> 20:07:18 [Noah]: dch sorry i will step back
> 20:07:26 [chewbranca]: Jarda: yeah definitely lots to be done
> 20:07:26 [dch]: :-) 
> 20:07:32 [jan____]: Jarda: this is a great place
> 20:07:32 [Noah]: i am in a reduced capacity - but also caffine saturated ;)
> 20:07:32 [benoitc]: so yeah viez sig,, is there anyrthoing to do
> 20:07:40 [benoitc]: how do e bench the new view server
> 20:07:41 [Noah]: ACTION steps slowly away from the keyboard
> 20:07:42 [dch]: ok let's rool
> # 2. 1.3.x blockers #
> 20:08:10 [benoitc]: would like also to discuss a little about the rcouch merge if ize
zill have some time
> 20:08:24 [dch]: #link is useful btw. Only 1654
> 20:08:25 [dch]: benoitc: noted!
> 20:08:25 [chewbranca]: dch: Fauxton is coming along, I'm hoping to get some feedback
on all-the-numbers and if there are any suggestions for better tests for views
> 20:08:47 [dch]: chewbranca: noted!
> 20:09:02 [rnewson]: I should add that davisp plans to do something on the view server
but he's not here to elaborate, sadly.
> 20:09:24 [dch]: so the guts of this is that the new view server in master/1.3.x requires
rebuilding your views as a result of changed file location and a different way of calculating
the sig.
> 20:09:40 [rnewson]: and the contents of the view file have changed in incompatible ways.
> 20:09:50 [rnewson]: just changing the sig and location is insufficient.
> 20:09:54 [dch]: davisp has a suspicion he did that for a reason but nobody recalls the
reason. It might just be an overdose of mustard-based bbq sauce though.
> 20:10:09 [dch]: right, just shuffling the view file from 1.2.x location into 1.3.x won't
work either.
> 20:10:17 [rnewson]: correct.
> 20:10:25 [benoitc]: i remeber it was to previent similarities in view 
> 20:10:32 [rnewson]: this is because the record name of the header has changed.
> 20:10:32 [benoitc]: s
> 20:10:39 [dch]: unless anybody has a miracle up their sleeve, rnewson and davisp have
this in hand
> 20:11:25 [dch]: presumably this means we need to keep the old view code around in some
form, so we can read the old views, until the next view compaction or regeneration?
> 20:11:39 [rnewson]: nah, it's easier than that.
> 20:11:39 [jan____]: I can help too but id need guideance.
> 20:11:40 [benoitc]: we need to make sure we wont't lost any new possible ffeature 
> 20:11:41 [dch]: ACTION phew
> 20:11:47 [benoitc]: whatever the change we do
> 20:11:49 [rnewson]: the question is whether we migrate to the new scheme or preserve
the old forever.
> 20:11:54 [dch]: sure
> 20:12:17 [rnewson]: I think we agree that the "Rebuild all views" approach is a non-starter.
> 20:12:24 [jan____]: +1
> 20:12:24 [benoitc]: or maybe we do like for dbs
> 20:12:26 [dch]: rnewson: i.e. if chewbranca's + others performance tests come back unfavourable,
then revert to 1.2.x approach?
> 20:12:39 [benoitc]: ie zork zith old sche;e and on compaction go to the new one
> 20:12:56 [jan____]: benoitc: Id like that approach
> 20:13:02 [dch]: in the bigger picture, one of the reasons for this change was to make
other indexing schemes more pluggable (e.g. vmx' R-tree, and anybody else game enouhg).
> 20:13:17 [dch]: benoitc: that seems sound.
> 20:13:18 [jan____]: bit I have not seriously looked at the code
> 20:13:24 [jan____]: *but
> 20:13:26 [benoitc]: it should be possible
> 20:13:39 [rnewson]: dch: imo, yes.
> 20:13:47 [chewbranca]: the very simple test I did comparing 1.2.x view engine vs 1.3.x
showed a minimal improvement for 1.3.x, but the test was far from exhaustive
> 20:13:47 [dch]: is the view server easier to extend, as a result? That would be a big
long-term win.
> 20:13:54 [benoitc]: imo the new view server design can introduce some latency but not
so much
> 20:14:03 [benoitc]: i noticed differences recently
> 20:14:18 [benoitc]: so that may be just a commit we did recently
> 20:14:25 [benoitc]: but we need benchs anyway
> 20:14:33 [jan____]: lets focus this discussion a little
> 20:14:34 [rnewson]: I wouldn't expect a performance difference btw, it just seems to
be an abstracting out of the common pattern.
> 20:14:40 [jan____]: can we move performance questions to the back?
> 20:14:57 [chewbranca]: benoitc: if you notice anywhere in particular where 1.3.x views
are slower, please give me a test case and sample data set and I'll add it to the benchmarks
> 20:15:18 [dch]: sure, let's keep on this for a few more minutes. jan____ anything specific
in mind?
> 20:16:03 [jan____]: rnewson: you have looked at the code, do you think it is feasible
to add some code that can treat a 1.2.x view index as is, and even write to it in the 1.2.x
format, and only upgrade to the new format on view compaction?
> 20:16:18 [jan____]: dch: I dont want to cut off anything, but Id like to not talk about
10 things at once
> 20:16:33 [benoitc]: chewbranca: well if you have already bench tools then share them
so e can have figures using the same process
> 20:16:34 [jan____]: we have a) file incompat, update badness b) various ways to handle
a) and c) performance stuff
> 20:17:10 [jan____]: c) might lead to a) being unnecessary for now (because wed revert)
but frankly I am not seeing that, and we need to address view index upgrades regardless)
> 20:17:20 [benoitc]: chewbranca: i actually can't share the data so i need to think a
little on how to test it accurately
> 20:18:03 [chewbranca]: benoitc:
> 20:18:05 [benoitc]: will do that but not before next week
> 20:18:13 [benoitc]: thanks.
> 20:19:19 [jan____]: I understand rnewson is out for dinner, davisp isnt here. is there
anyone else who can comment on whether the proposed strategy would work?
> 20:19:25 [dch]: so (a) is all about the view signatures themselves, as a result of the
changed data format? I'm not clear on the issue, but it sounds like rnewson is saying the
new sig is a requirement of the new view engine, we can't keep the old sig?
> 20:19:55 [benoitc]: dch: the thing is that you have a big index you don't want to rebuild
it online 
> 20:20:13 [jan____]: dch: it is the filename and the file format
> 20:20:33 [dch]: benoitc: clear, you'd want to do the COPY shuffle for example, or plan
your compaction.
> 20:20:41 [benoitc]: dch: yup
> 20:20:48 [jan____]: that leads to 1.3.x (as I understand) invalidate any previous view
indexes which leads to what benoitc just said
> 20:20:55 [wendall911]: jan____: not having support for both 1.2.x and 1.3.x view formats
would mean downtime. Huge downtime for some cases.
> 20:21:03 [benoitc]: for old sig vs new sig, i can see that the new signature make them
more unique
> 20:21:10 [dch]: yes, and there's no way to avoid it either.
> 20:21:10 [jan____]: wendall911: yes, we want to avoid that
> 20:21:28 [jan____]: we can defer it to view-compaction though
> 20:21:39 [kocolosk]: I'd have to think we can add some upgrade clauses to read the old
signature format internally (though i've not looked closely at it yet)
> 20:21:39 [benoitc]: i think we need for that some feedback from davisp
> 20:21:54 [jan____]: wel *can* modulo code that the db file handling already has (not
that we can copy anything I think)
> 20:22:02 [kocolosk]: the change in location is a bit more annoying but should be manageable,
> 20:22:02 [jan____]: *well
> 20:22:17 [benoitc]: kocolosk: yes should be possible, need to check if it works well
with the server that maintain states
> 20:22:24 [kocolosk]: right
> 20:22:39 [benoitc]: kocolosk: the location change is easier to solve imo
> 20:22:47 [jan____]: kocolosk: wanna take a stab at it?
> 20:22:54 [kocolosk]: try new location, if missing try old one?
> 20:23:33 [kocolosk]: jan____: if i can't scare up davisp, yeah
> 20:24:03 [jan____]: kocolosk: woot, I am up for testing this tomorrow, if you can tag-team
this over night
> 20:24:10 [benoitc]: kocolosk: yes that can work 
> 20:24:25 [kocolosk]: jan____: kk
> 20:24:56 [dch]: is this then per-db & view, or per view? and how will we know that
all the views are completely migrated/updated, and we can transfer the old ones to .delete
for example?
> 20:25:12 [dch]: ACTION thinking out loud, not expecting an answer.
> 20:25:33 [benoitc]: chewbranca: i think this test lacks of some datasets types (size
of docs and real latency) i will try it anyway
> 20:26:10 [benoitc]: dch: can be done when accessing the view group maybe
> 20:26:27 [chewbranca]: benoitc: definitely does, just put it together over the weekend,
lets chat later in this meeting about what can be done to make it comprehensive 
> 20:26:48 [dch]: ok I think we are ready to move to next topic, unless people have more
to add?
> 20:27:06 [dch]: anything else people want to add for 1.3.x issues?
> 20:27:11 [benoitc]: chewbranca: fine for me
> 20:27:19 [benoitc]: let's try tonmorrow though
> 20:27:27 [benoitc]: i'm a bit toired tonight
> 20:27:48 [benoitc]: dch: the doc link maybe
> 20:27:55 [benoitc]: in fiuton
> 20:28:04 [jan____]: one more point for the release team: this might push us over the
desired Feb 1st call-a-vote date.
> 20:28:05 [dch]: I have one, I added a jira a while back to do sanity checking that (a)
we can read/write to key locations like logfiles, view and dbs, and if not, fail with a graceful(ish)
erlang message. I will take a whack at this tomorrow, it will not be a lot of code
> 20:28:10 [benoitc]: maybe if docs aren't build we can provide direct link to readthe
> 20:28:33 [dch]: benoitc: I am +1 on a direct link to if the docs aren't
> 20:28:48 [jan____]: I liked the idae of doing the fake index.html with the meta refresh
> 20:28:48 [dch]: I haven't looked at Kxepal's patch yet.
> 20:29:06 [jan____]: neither, dch wanna do that tomorrow?
> 20:29:19 [dch]: jan____: read the patch? for sure.
> 20:29:39 [benoitc]: dch: mm i would just display it, misued the word direct
> 20:29:42 [benoitc]: misused
> 20:29:49 [benoitc]: ACTION is training a new kbd
> 20:30:32 [dch]: ok, next up is.
> # 3. rcouch merge #
> 20:30:40 [dch]: benoitc: all yours
> 20:30:54 [rnewson]: jan____: what kocolosk said.
> 20:31:04 [jan____]: rnewson: thx
> 20:31:47 [benoitc]: ok 
> 20:31:54 [benoitc]: so i am thinking to start it next week
> 20:32:09 [benoitc]: by first introducing rebar and a way to build without autoconf
> 20:32:24 [benoitc]: i won't try to fight with autotools for now
> 20:32:32 [benoitc]: except if one think it's desired
> 20:33:02 [dch]: benoitc: good call. and then work on the NIFs? I should be able to help
with that.
> 20:33:09 [benoitc]: yes
> 20:33:18 [benoitc]: i can say i have already all the nifs in rcouch 
> 20:33:32 [benoitc]: couch_collate for ex
> 20:33:50 [dch]: yup, that should be pretty good. I think I forgot one piece in jiffy
for windows, but I assume we are sticking with ejson for the moment?
> 20:33:54 [benoitc]: wich replace couch_drv & couch_ejson_compare
> 20:34:09 [benoitc]: un didn't thin k baout that
> 20:34:24 [benoitc]: yes so i will put this change in another commit
> 20:34:32 [jan____]: dch: yeah, lets not make it more comlpicated than needed
> 20:34:34 [jan____]: we can jiffy things in at any time later
> 20:34:47 [benoitc]: going to jiffy is a 5 line patch anyway
> 20:35:09 [jan____]: yes, but it adds complexity for all the systems we support
> 20:35:17 [benoitc]: i agree
> 20:35:24 [jan____]: cool
> 20:35:47 [dch]: alright! sounds great, so what was actiomnable there?
> 20:35:54 [benoitc]: i am not sure how to handle the spidermonkey case for now
> 20:36:09 [dch]: #action benoitc working on introducting rebar to couch, as part of rcouch
> 20:36:32 [dch]: I think for spidermonkey we should seriously consider consolidating on
1.8.5 only.
> 20:36:32 [benoitc]: yes let put an action, what version should be supported
> 20:36:39 [benoitc]: presumably we donyt replace everything for now
> 20:36:54 [benoitc]: yes i'm +1 on it
> 20:37:02 [dch]: for 1.3.x, no, this would come 1.4 or later not sure how that is covered
in semver.
> 20:37:04 [jan____]: for master? +1
> 20:37:17 [benoitc]: yes for master
> 20:37:32 [dch]: it's very S&M tonight
> 20:37:41 [dch]: benoitc: anything else, or shall we move topic?
> 20:37:56 [benoitc]: i think i have some answer
> 20:38:09 [benoitc]: will make spome update on the ml next week
> 20:38:17 [jan____]: cool
> 20:38:19 [benoitc]: ACTION is disappearing for the fosdem
> 20:38:24 [dch]: looking forwards to it!
> 20:38:32 [dch]: have fun benoitc.
> # 4. fauxon, bring me all your UX goodness. #
> 20:38:49 [dch]: drsm79: how does your 3G go?
> 20:39:04 [dch]: or chewbranca maybe it was your topic (sorry!)
> 20:39:32 [chewbranca]: fauxton is coming along well, plugin system is solid and we've
been getting some help from Garren and a few others
> 20:40:09 [chewbranca]: we're in the process of getting CLAs taken care of so that we
can move the code into the primary repo and use standard ticketing tools and what not
> 20:40:11 [dch]: great! are we ready to set up an integration branch for this or are you
still githubbing?
> 20:40:24 [jan____]: Simon got @gr2m interested in hacking too
> 20:40:33 [jan____]: I want to move the code to asf asap
> 20:40:39 [dch]: chewbranca: right. let me know if you need a hand with this, the actual
process is easy.
> 20:40:47 [jan____]: Ill handle the apperwork, I hope I get this done tomorrow
> 20:40:47 [drsm79]: dch what chewbranca said + more hands welcome
> 20:40:47 [jan____]: *paper
> 20:40:54 [drsm79]: dch:  it's on a branch in a fork of couch
> 20:40:54 [drsm79]: we can switch it over whenever I think
> 20:41:02 [jan____]: then I want to kick off votes for all fauxton committers
> 20:41:04 [dch]: fun-tastic!
> 20:41:10 [chewbranca]: plenty left to do, will have a view editor working this week that
would be good to get some feedback
> 20:41:32 [chewbranca]: yeah jan____ is helping getting the paperwork taken care of, I'll
get my CLA signed and scanned and sent off to jan____ today
> 20:41:40 [drsm79]: I'll do mine tomorrow
> 20:41:47 [jan____]: also, @gr2m is a little frustrated with fauxton development. I asked
him what hed like it to look at and he wrote us up a fake readme:
> 20:41:47 [jan____]: oops
> 20:41:56 [jan____]: that fake readme is actually in
> 20:42:24 [drsm79]: 3g is failing me
> 20:42:47 [drsm79]: will read it 
> 20:42:54 [chewbranca]: jan____: sure, things could be easier, and we just need to throw
deps into a package.json and then you can just npm install
> 20:42:54 [Jarda]: jan____: that looks awesome to me
> 20:43:09 [jan____]: Jarda: yeah, sadly it isnt that for now
> 20:43:33 [chewbranca]: I'll work on simplifying the setup process a bit this week
> 20:43:40 [jan____]: but I think we can approximate things
> 20:43:47 [benoitc]: mmm btw does npp or other can keep deps offline ?
> 20:43:49 [dch]: #action ASF CLAs underway for fauxtoneers
> 20:43:54 [jan____]: I wonder how folks feel about a separate repo for fauxton?
> 20:43:55 [benoitc]: for a n archive release i mean
> 20:43:56 [dch]: benoitc:  yes.
> 20:44:09 [daleharvey_]: I think a seperate repo is a good idea
> 20:44:12 [dch]: benoitc: yes you can in principle commit them.
> 20:44:17 [benoitc]: also everything can be solved later during full integraton
> 20:44:17 [Jarda]: jan____: so what is the tech stack for current fauxton?
> 20:44:24 [jan____]: chewbranca: awesome, Im happy to help test and nudge towards MOAR
EASY so we can get the gr2m types aboard.
> 20:44:24 [benoitc]: dch: mm not commit
> 20:44:39 [benoitc]: or maybe
> 20:44:54 [jan____]: Jarda: chewbranca knows ;)
> 20:45:04 [chewbranca]: Jarda: its a backbone app with grunt for building/tasks and a
handful of other tools like backbone layoutmanager
> 20:45:09 [Jarda]: I have a solid background in client-side javascript (backbone/requirejs/etc)
> 20:45:11 [benoitc]: we need to integrate sonme depenqncies on release but don't really
need all in the repo
> 20:45:24 [Jarda]: We also use yeoman in our current project at work
> 20:45:39 [chewbranca]: current code is at:
> 20:45:54 [Jarda]: I might be able to help if there are clear issues to tackle
> 20:45:57 [jan____]: Jarda: perfect :)
> 20:46:02 [dch]: Jarda: feel free to pitch in, we'd love to have some helping hands!
> 20:46:39 [drsm79]: jan____: I'd be happy to have it in it's own repo etc
> 20:46:40 [drsm79]: but don't know all the nitty gritty around what ASF require
> 20:46:54 [jan____]: just admin stuff
> 20:46:57 [jan____]: we can have more repos
> 20:47:02 [drsm79]: moar
> 20:47:09 [dch]: ok, moar fauxton or next topic?
> 20:47:10 [jan____]: the only thing we need to solve then is how to get it into `make
> 20:47:17 [drsm79]: do issues have to go in jira or can we do it all in github?
> 20:47:17 [jan____]: moarepos
> 20:47:17 [dch]: chewbranca: drsm79: great update.
> 20:47:32 [chewbranca]: would be good to decide on a place to handle tickets
> 20:47:54 [benoitc]: jira once the integration is done i think
> 20:47:57 [drsm79]: dch: not bad considering being on a train ;)
> 20:48:04 [jan____]: drsm79: once code is in ASF, issue go into jira, PRs can go into
> 20:48:09 [chewbranca]: grunt is similar to yeoman, we can get down to a npm install grunt
gogo-gadget-fauxton and have it all running, but would be good to have a place where we can
start recording issues
> 20:48:18 [chewbranca]: dch: thanks
> 20:48:18 [jan____]: Id suggest abusing PRs for tickets
> 20:48:32 [chewbranca]: jan____: that's what we've been doing
> 20:48:32 [jan____]: tickets proper are ASF JIRA for now
> 20:48:47 [jan____]: good
> 20:48:49 [jan____]: keep it up :)
> 20:49:02 [drsm79]: jan____: sounds good
> 20:49:11 [jan____]: new PRs would go against or whatever
> 20:49:24 [Jarda]: ok so is there a roadmap for fauxton somewhere?
> 20:49:39 [chewbranca]: should we temporarily have issues at the current fauxton repo
or in the futon organization?
> 20:49:56 [jan____]: chewbranca: just to migrate things a week later?
> 20:50:02 [chewbranca]: we're starting to get enough people involved that we need somewhere
for people to file issues
> 20:50:09 [jan____]: chewbranca: we can start recording issues in ASF jira right away
> 20:50:11 [chewbranca]: jan____: good point, alright scratch that
> 20:50:24 [benoitc]: ACTION dch ping here
> 20:50:24 [benoitc]: err
> 20:50:32 [Jarda]: if one would like to help, is the proper way to just make it running
and hack around?
> 20:50:39 [benoitc]: z\/win 59
> 20:50:40 [chewbranca]: jan____: sounds good
> 20:50:54 [dch]: next topic?
> 20:50:55 [jan____]: chewbranca: aye
> 20:50:57 [chewbranca]: Jarda: yeah, shoot me a ping after this and I can help you get
up and running
> 20:51:02 [Jarda]: chewbranca: cool
> 20:51:04 [jan____]: #action jan to add fauxton component to JIRA
> 20:51:09 [jan____]: +1 EOD
> 20:51:32 [dch]: ok I wanted to circle back to 1.3.x / CORS for a moment..
> # 5. 1.3.x CORS #
> 20:52:02 [dch]: basically I did some testing, and I wondered if we collectively feel
ok or if we should be doing some more here?
> 20:52:32 [benoitc]: ah more like what?
> 20:52:39 [jan____]: dch: we ship as experimental
> 20:52:40 [dch]: I got stuck reading some of the spec and translating that into "testable"
> 20:52:56 [jan____]: dch: I dont want to reopen this now. we dont get to ship 1.3.x ever
> 20:52:56 [dch]: pre-flight for example, I didn't follow that sufficiently yet.
> 20:53:09 [dch]: jan____: ok for me then.
> 20:53:11 [benoitc]: we are ok on that imo 
> 20:53:32 [benoitc]: also i would say that having pouchdb running with it is enoiugh for
me :)
> 20:53:32 [dch]: woot, I will see if I can put together an example or two for people to
> 20:53:34 [daleharvey_]: trying to find my old comments, but I remember there being a
few minor feature requests, but mostly good, ship as is experimental and then to be honest
there was only a few things more needed
> 20:53:47 [jan____]: yeah, lets have it hit eager users and let them find all the issues
> 20:53:54 [dch]: yay users!
> 20:53:56 [jan____]: daleharvey_: I think I got through mot of your stuff eventually
> 20:54:03 [dch]: ok, next topic, any requests?
> 20:54:24 [chewbranca]: benchmarking?
> 20:54:34 [jan____]: I can give an update about CouchDB Conf
> 20:55:04 [dch]: um how about we roll with jan____ then?
> 20:55:04 [benoitc]: chewbranca: i need to go , as qction i would say let's try together
to build some datasets 
> 20:55:09 [Noah]: jan____: to the ML preferably
> 20:55:09 [benoitc]: so we can test it
> 20:55:09 [chewbranca]: yeah would be great to hear about CouchDB conf
> 20:55:18 [drsm79]: jan____: we should start planning the next one ~now
> 20:55:18 [chewbranca]: benoitc: sounds good
> 20:55:18 [drsm79]: keep the momentum 
> # 6. couchdbconf was awesome #
> 20:55:26 [jan____]: Noah: aye
> 20:55:26 [Noah]: cool
> 20:55:26 [jan____]: ok
> 20:55:34 [jan____]: #action jan to mail about CouchDB Conf
> 20:55:42 [jan____]: drsm79: +1
> 20:55:49 [chewbranca]: I'm definitely interested in getting a CouchDB conference going
out in Seattle
> 20:55:58 [drsm79]: chewbranca in ~4 months time?
> 20:56:06 [chewbranca]: yeap
> 20:56:12 [drsm79]: gtg train has got home
> 20:56:19 [jan____]: drsm79: safe home!
> 20:56:27 [chewbranca]: drsm79: later
> 20:56:27 [jan____]: chewbranca: yeah, awesome
> 20:56:52 [jan____]: chewbranca: for me around JSConf US or NodeConf
> 20:57:04 [jan____]: otherwise I can skype in
> 20:57:25 [chewbranca]: jan____: sounds good
> 20:58:09 [dch]: ok we have a choice, run over for 5-10 minutes and talk about benchmarking
or do a sprint for the finish.
> 20:58:47 [jan____]: Im all actioned out, but you go ahead
> 20:58:47 [dch]: chewbranca: any actions related to seattlecouch?
> # 7. benchmarking #
> 20:59:32 [dch]: #link
> 20:59:32 [chewbranca]: dch: if it sounds good to everyone I'll start looking into it
> 20:59:56 [chewbranca]: nice, good link
> 21:00:17 [jan____]: nice link!
> 21:00:24 [jan____]: yeah! :)
> 21:00:24 [dch]: chewbranca: I would love to do some more work with what Adam put together
in Tsung, its on my list for a while. But I need to keep focused on the other stuff first.
> 21:01:17 [dch]: Also I use this a lot its ruby,
set it up with a while true; do  and run multiple instances pointing to different DBs. This
is a nice testing approach.
> 21:01:17 [chewbranca]: so my idea for all-the-numbers is to get a simple view benchmark
in place that will easily compare vanilla spidermonkey view engine, with Jason's node.js engine,
and then also compare a native erlang view engine, to get some ideas on overall performance
and cost paid for serialization
> 21:01:32 [chewbranca]: I've got this going:
> 21:01:39 [dch]: #link
> 21:01:54 [chewbranca]: which is not Tsung by any means, but works ok and saves the results
back into couchdb for easy comparisons
> 21:02:24 [chewbranca]: not opposed to using something else, but I want to understand
the performance characteristics of the different options, and I think it would be good to
get that information out
> 21:02:40 [dch]: chewbranca: great, the view engine changes are a good place to start.
> 21:03:00 [chewbranca]: dch: yeah I sent out an email to dev@ comparing 1.2.x with 1.3.x
> 21:03:01 [dch]: there's also a python view server, I think kxepal wrote it ?
> 21:03:38 [chewbranca]: I can take care of adding a few of the different engines, but
what I could use help with is determining the appropriate sample data sets and proper views
to comprehensively compare the engines
> 21:04:02 [dch]: chewbranca: tricky to get real-world stuff. Um I know a few people with
large docs and complex views, maybe they can share stuff.
> 21:04:23 [dch]: #action dch to find some people with large docs/complex views for testing
> 21:04:45 [chewbranca]: dch: agreed, if there are some known edge cases it would be good
to get those in
> 21:05:08 [chewbranca]: I want to have a range of doc sizes and what not so you can compare
serialization cost as a function of doc size and number of views per ddoc and what not
> 21:05:23 [Kxepal]: dch: you want to benchmark it too?
> 21:05:23 [dch]: chewbranca: a point made a while back is that  testing should ideally
be done with concurrent runs. That doesn't need to be phase 1 but its good to add it in future.
> 21:05:41 [dch]: Kxepal: more to the point, are you interested in helping out and giving
it a test too ? 
> 21:05:54 [dch]: you know how it fits together so you'd be a great help :)
> 21:06:08 [chewbranca]: dch: yeah I manually did 3 runs for the tests, but I'm planning
on adding automatic iterations 
> 21:07:10 [wendall911]: it isn't always ideal to run concurrently, especially building
large indexes. You may just get to the point where you're actually just benchmarking the hardware,
not the software.
> 21:07:39 [Kxepal]: dch: no problem. however, for python there is some set of cases when
it will be faster due to rich stdlib and caching imports. but mostly it's slower on long running
tasks unless it's not a pypy
> 21:08:18 [chewbranca]: what other concurrent requests would be good to do? basically
right now I'm just throwing a bunch of docs at the db and then grabbing the view with limit=1
> 21:08:25 [dch]: chewbranca: I'd like to run this on windows as well, um maybe I send
you a patch to extract the OS etc for the reports.
> 21:08:25 [chewbranca]: not testing incremental updates or anything else right now
> 21:08:33 [chewbranca]: dch: sounds good!
> 21:08:49 [dch]: compaction
> 21:09:33 [chewbranca]: compaction as in bench marking compaction as well, or running
compaction while bench marking the view?
> 21:09:54 [wendall911]: Is it possible for the test runner to use tmpfs for the databases?
I fear that you'd be just doing IO testing on the hardware for some of this.
> 21:09:56 [dch]: e.g. run tests, compact, re-run tests.
> 21:09:57 [chewbranca]: maybe action item for this is start a thread on the ML about proper
benchmarking scenarios and start adding those in
> 21:10:17 [dch]: chewbranca: yes, or adding them into a jira ticket list.
> 21:10:30 [chewbranca]: dch: ahhh, yeah I was just loading the db, deleting, reloading
the db
> 21:10:32 [dch]: wendall911: I think we probably want both.
> 21:10:53 [wendall911]: dch: I don't see that figuring out your hardware is slow is really
a useful benchmark
> 21:11:00 [dch]: I definitely want to see difference between SSD and cloudy, and real
> 21:11:08 [wendall911]: hmm, ok
> 21:11:31 [dch]: wendall911: because for example the serialisation cost is high, possibly
comparable in some circumstances to disk IO slowness. 
> 21:11:44 [dch]: rampant speculation that.
> 21:11:59 [wendall911]: so long as the benchmarks are run on hardware that the test is
the only thing running
> 21:12:01 [dch]: but I've seen 650K individual docs going to the JS view server.
> 21:12:07 [chewbranca]: understanding the actual serialization costs was one of my big
motivations here
> 21:12:07 [dch]: wendall911: yup.
> 21:12:15 [dch]: chewbranca: I have some ideas on testing serialisation costs.
> 21:12:45 [chewbranca]: dch: cool, definitely want to pick your brain on it
> 21:12:50 [dch]: I can instrument erlang and node directly (ignoring sandboxes) and we
should be able to get fine-grained results.
> 21:12:57 [chewbranca]: nice
> 21:13:12 [dch]: but thats a wee bit away. I have the basic pieces working but not massaging
the data.
> 21:13:19 [wendall911]: and we should only publish the tmpfs without fsync, so we can
be like all the other kids
> 21:13:27 [dch]: wendall911: noatime FTW.
> 21:13:34 [wendall911]: :)
> 21:13:57 [dch]: chewbranca: if you're feeling frisky we could add with / without snappy
> 21:14:12 [chewbranca]: dch: yeah the more tests we can do the merrier
> 21:14:19 [dch]: there must be a generic "run tests with various combinations" nodejs
library out there.
> 21:14:49 [chewbranca]: yeah worth looking into
> 21:15:12 [dch]: so anything more? I'm getting calls to come to bed.
> 21:15:14 [dch]: (sorry for the abrupt topic change)
> 21:15:20 [chewbranca]: nope, glad to get the discussion going
> 21:15:50 [Kxepal]: chewbranca: if you'd like I could share db for 3M of production-world
documents for testing - around 6GB of data and they are complex to create interesting views
that really maps and reduces.
> 21:15:57 [dch]: yes, I am totally enthused.
> 21:16:32 [dch]: Kxepal: that would be *awesome*. I am happy to sanitise the content,
and then we could potentially keep it as reference, if you agree. 
> 21:16:39 [dch]: and maybe we could rewrite some of the views in erlang.
> 21:16:47 [dch]: for comparison.
> 21:16:55 [wendall911]: dch: is there a place to have a hosted copy? Would be nice to
be able to run tests without distributing the data per-se, just sync with hosted copy.
> 21:17:02 [chewbranca]: Kxepal: yeah that would be great, especially if we can throw that
up on a CouchDB instance somewhere so people can replicate it in for testing
> 21:17:24 [Kxepal]: dch: simple erlang view that build index for single field runs half
hour for my server (: never tried python nor js for this db
> 21:17:25 [dch]: wendall911: I am pretty sure between cloudant and iriscouch we will find
a way. Or use my mac as an endpoint.
> 21:17:39 [wendall911]: haha
> 21:18:04 [Kxepal]: chewbranca: ok, I'll prepare db on this week. ping me if I didn't
> 21:18:09 [dch]: wendall911: what do you mean by hosted copy? a way to retrieve the doc
set for testing?
> 21:18:17 [wendall911]: dch: yes
> 21:18:24 [chewbranca]: yeah worst case scenario I've got a dedicated box with couchdb
running and plenty of bandwidth
> 21:18:26 [dch]: #action kxepal to contribute real world docs and views for testing -
huge thanks!
> 21:18:46 [dch]: chewbranca: like a spare 80 node cluster I'll bet. 
> 21:18:53 [dch]: ok any more topics?
> 21:19:00 [dch]: we covered a lot
> 21:19:01 [chewbranca]: lol
> 21:19:23 [wendall911]: On a side note, I may be able to step into a co-maintainer role
on the epel packages (for centos, rhel, fedora)
> 21:19:31 [wendall911]: working on it
> 21:20:08 [wendall911]: But not really project related, just frustrated with the slowness
of releases downstream, so proactively trying to fix.
> 21:20:18 [dch]: wendall911: that would be fantastic, we are working on build bots (via
jenkins etc) so you might be interested in that too.
> 21:20:44 [wendall911]: ahh, would be nice. I think the show stopper for me is js 1.7
on centos, it just sucks
> # 8. packages #
> 21:21:22 [dch]: #action wendall911 to come back to ML about EPEL packages
> 21:21:30 [dch]: wendall911: hopefully that's generic enough to remind you.
> 21:21:58 [wendall911]: dch: thanks, I would have done more in this area, just work is
overly busy atm :(
> 21:22:12 [dch]: wendall911: damn that work.
> 21:22:42 [dch]: imma closing the meeting unless we have more?

View raw message