Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.214.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABvT1DGRnsZqMtVtzv7RUSnoxV79GRAZdb0tvWcJr8i96gxrMQ@mail.gmail.com>
References: 
 <CAPaJBx7vBO9A_NSNepfoCLwd9gUGXrapjFTDBR=13NsBjiPDMA@mail.gmail.com>
 <CAKmKYaBbL3bjHFwsBkiEtaCiAL-jTTaD=fui9=CZdP7PRHJMwg@mail.gmail.com>
 <CAPaJBx6Fob+nRBaqGSnOHgS3cvrx2OLB_edtFQ-LH=VdnTvWKg@mail.gmail.com>
 <CABvT1DGRnsZqMtVtzv7RUSnoxV79GRAZdb0tvWcJr8i96gxrMQ@mail.gmail.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Thu, 7 Mar 2013 12:02:04 -0600
Message-ID: 
 <CAJ_m3YBve5+-cVLr7-a4sL1dwFMsG2ZDSCpXS=vi=V-G5OxO_A@mail.gmail.com>
Subject: Re: [DISCUSS] Goals for 2013
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

There's a lot of work that could be done to a lot of the internals of
CouchDB to clean things up and put us back in a place where we can be
quicker to innovate.

1. Rewrite the HTTP layer to not be a gobbledygook function call
spiderweb for request dispatching. Whether we use something like
webmachine or switch to Cowboy or whatever it takes. The HTTP layer is
entirely too complicated for what it does.

2. Cleanup the core database implementation. There's a lot of cruft
that's grown up in some of the core code that makes things
unreasonably hard to follow. Once you untangle the spaghetti you
realize that CouchDB is exceedingly simple, but not many people make
it through that initial learning investment. This also ties into #3.

3. Analyze hot code paths for dumbness to improve database
performance. Part of the code being complicated is that its really
hard to tell whats going on at various times. I've traced through
different parts debugging various things and have noticed a number of
lackluster behaviors (things like re-reading the btree root multiple
times for a single document update). Someone with enough interest
could start tracing through here and counting the ways we screw the
pooch on this count. I'm also still convinced that something in the
HTTP layer is an artificial bottleneck but haven't had time to
investigate.

4. Rewrite couchjs and the view server protocol. We've talked about
this for ages. Moving from multiple OS processes to a single process
that handles all requests for a given language asynchronously and
handles caching could turn into a significant performance benefit for
lots of database operations.

5. Rewrite the entire test suite. From scratch. In a way that doesn't
break things. There are some interesting ways to approach this. I've
played before with Erlang/Python hybrids for doing internal/external
testing. Nothing quite congealed into a cohesive platform but I'm
convinced that our current approach is lackluster at best.

6. Other things we've previously discussed. Rewrite the replicator to
focus on robustness. Change conflict beahviour so they can't be
ignored easily. Split couchapp stuff into a different project to
reinforce the separation of concerns. Remove metadata from doc bodies.
All the 2.0-y things.

On Thu, Mar 7, 2013 at 10:23 AM, Robert Newson <rnewson@apache.org> wrote:
> +1. We are conscious of our release process failings but it's great to
> see an uninhibited reminder like this.
>
> We're aiming for regular, frequent releases after this one, as Noah
> says. I want to see that process so regular that it becomes boring.
> You should come to expect that a fix is out within a month and new
> features are out within a quarter.
>
> We are also making some excellent progress at Cloudant preparing the
> BigCouch merger. We hope to have something for folks to play with by
> the end of next week. I'll also add that when I say "BigCouch", I'm
> referring to the well-tendered version at Cloudant and not the 0.4 we
> cut over a year ago. This is *not* stale code, but the very well
> cared-for code we run in production. We'll have updates as major
> pieces come but we (Paul Davis and myself) are largely heads-down
> dealing with all kinds of lovely merge conflicts and stuff. :D
>
> B
>
> On 7 March 2013 09:54, Noah Slater <nslater@apache.org> wrote:
>> Agreed. Thanks for the honest feedback. Massively appreciated.
>>
>> I have been very frustrated with the 1.3.0 release. It was, and still is=
,
>> supposed to be the first of our regular releases. Unfortunately, the
>> release was mired with a raft of CVEs that came first, and then some tho=
rny
>> blockers. Not making excuses. I felt the pain of this as much as anybody
>> here who cares about making this work.
>>
>> I've spent the last few weeks putting work in, and actually coming up wi=
th
>> an entirely new release procedure (which I will be announcing in tandem
>> with the release, in a day or two) that should address these problems. M=
ost
>> significantly, I am moving the test suite to the voting stage, which wil=
l
>> allow me to cut and call votes very quickly.
>>
>> So, from my perspective, we are just about to emerge from release hell, =
and
>> I have my eye firmly on the next three quarters. As I mention in my
>> original email, basically, most of this hinges on activity. And activity
>> hinges on releases, as you say. But I've already put a lot of thought in=
to
>> that, and I wanted to invite the community to start thinking about the
>> other things we can start to address. I think these things are important
>> too.
>>
>> Here's an example, look at the burn-down chart for tickets:
>>
>> https://issues.apache.org/jira/browse/CouchDB
>>
>> Abysmal.
>>
>> How do we solve it? Sure, release will stimulate activity. But again, th=
is
>> is a known quantity. And I hope. Oh God, I hope. That we're going to nai=
l
>> that one. So I am left asking, from my POV, what else?
>>
>> Well, the stuff I put in my email, I guess. I have a big backlog of proj=
ect
>> management stuff to get through. Things I want to bring to the list. Thi=
ngs
>> that should hopefully get easier as activity picks up.
>>
>> Perhaps some of these are things you might be interested in? One of the
>> things we really really desperately need is someone to organise and run =
a
>> JIRA triage team. The goal being, I think, to just keep an eye on JIRA,
>> field incoming tickets, close them out where appropriate, or categorise =
and
>> assign them otherwise.
>>
>> We also need a docs team. And a marketing team. And a PM team. These tea=
ms
>> might just be one person to begin with. But I think we need someone to s=
tep
>> up to the plate and say "I will look after this."
>>
>> There are lots of ways to contribute that don't involve Erlang. I don't
>> know Erlang. :)
>>
>> Again, I think some metrics would be a good idea. For example, if we say=
,
>> okay, well, we want to double our committership by Q4. That invites a
>> conversation. How do we do that?
>>
>> These are important questions to be asking!
>>
>>
>> On 7 March 2013 11:04, Dirkjan Ochtman <dirkjan@ochtman.nl> wrote:
>>
>>> (warning, grumpy rant forthcoming...)
>>>
>>> On Wed, Mar 6, 2013 at 7:14 PM, Noah Slater <nslater@apache.org> wrote:
>>> > Thoughts?
>>>
>>> It seems to me that a lot of this hinges on releases. Releases
>>> generate publicity and therefore developer interest, thus developer
>>> engagement. Releases make sure that work developers do gets into
>>> user's hands soon.
>>>
>>> CouchDB, and I am sorry to have been banging this drum for a few years
>>> now, is the worst open source project at doing releases I know.
>>>
>>> Here's a quote from this list, from February 13:
>>>
>>> > I plan to cut a release candidate from the 1.3.x branch this weekend.
>>>
>>> And another, from Nov 13:
>>>
>>> > when we set out to ship 1.3.0, we thought the cors and docs
>>> > branches were just around the corner. That was a couple of
>>> > weeks ago. We are just starting with this, but the whole
>>> > idea of time-based releases is that we do not wait for
>>> > feature branches to be ready.
>>> >
>>> > I=92d like to propose that we ignore everything we=92ve said
>>> > before and do the following:
>>> >
>>> >  - ship 1.3.0 as reflected in the 1.3.x branch now.
>>>
>>> Here's another, from October 23:
>>>
>>> > It's been a while since we released and I want to change this now. I
>>> > propose making a 1.3.x branch from today's master
>>> > (66529378dd06342929e04772370f3509cbe786a5) and building 1.3.0 there.
>>>
>>> Another, from June 16:
>>>
>>> > 1. We'd like to proposed formal time-based releases
>>>
>>> Et cetera. As an example, the EventSource bits, which are actually
>>> something which would be very useful for us at work, were committed to
>>> the tree on May 16. They are yet to be released.
>>>
>>> I feel a bit hypocritical for saying all this, since I don't
>>> contribute much to core CouchDB (although I did do a bunch of the doc
>>> conversion, try to be helpful around the edges and maintain
>>> couchdb-python). The reason for this is mostly that I thoroughly
>>> dislike Erlang as a language, and have no intention of learning it
>>> just to help out with CouchDB. I did some Futon work back in the day,
>>> but had lots of trouble getting it committed since no one's really
>>> owned Futon for a while (and right now, it looks like other people
>>> have that bit covered, though they're using a modern but also quite
>>> complex development stack that will make it unlikely for a lowly
>>> back-end web developer dude like me to be able to contribute much).
>>>
>>> So, IMNSH and annoyingly harsh O, stop talking so much about how to
>>> energize the project, and Just. Ship. It. Every month, preferably. Fix
>>> the test suite and whatever else makes it so damn hard to release.
>>> Insert story here about that dude who realized people were trying to
>>> fix the wrong problem [1].
>>>
>>> End of rant, cheers,
>>>
>>> Dirkjan
>>>
>>> [1] http://www.azarask.in/blog/post/the-wrong-problem/
>>>
>>
>>
>>
>> --
>> NS