Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.160.180 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=XgDnWJYehodu/Mzmg/rX7xqBRQVMEJfDGuzX1XR6+gpleS2WAwS9xHEU+5wDfUFYd5
         IE69+p8HrWiIB6pyHjv8mjiqoxA4FjfJLmuXQ5iBEe4hB4bQhNg3fE89MWs2M7wQdjLP
         Jz7CXXNG2/jZ5d5c9bQeOtnVJeSbJkDw4dHzk=
MIME-Version: 1.0
In-Reply-To: <AANLkTimhpY2HXtAoPtRwRzHsaVfQkYzG70Yt3QbKefJ_@mail.gmail.com>
References: <AANLkTimNk_96tNkZC57WYMg-u8=1Aa30J6dpZSN0RsMv@mail.gmail.com>
 <AANLkTim8kqN9knN5Y7_5Q3pCEJ8D44yQykHF6GWb+nv1@mail.gmail.com>
 <F796ABEF-828F-428C-B67E-25A4B40D7621@apache.org>
 <AANLkTikZsv=-4FJird+-s6xLx6Et+P_nFKW=rW6yTx7e@mail.gmail.com>
 <791E5E22-088B-424C-83BF-506FECA8D9F3@apache.org>
 <AANLkTimhpY2HXtAoPtRwRzHsaVfQkYzG70Yt3QbKefJ_@mail.gmail.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Wed, 3 Nov 2010 23:27:05 -0400
Message-ID: <AANLkTimRdTUV0YyM4FpUwfv+Zh3PbevSncZnj2-rNwMJ@mail.gmail.com>
Subject: Re: CouchDB OTP
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Nov 3, 2010 at 8:45 PM, Tristan Sloughter
<tristan.sloughter@gmail.com> wrote:
> Agreed. That sounds like a good plan. I'd just want to ensure that the
> Erlang side can be installed as a release and run as a release, or includ=
ed
> as apps, to a project and run without a problem. This can be complicated =
by
> having the build system do so much with the configuration files. Which is
> why I simply put them in priv and reference them that way. But something =
can
> be figured out.
>
> Tristan
>
> On Wed, Nov 3, 2010 at 7:40 PM, Noah Slater <nslater@apache.org> wrote:
>
>>
>> On 4 Nov 2010, at 00:33, Tristan Sloughter wrote:
>>
>> > What complex stuff is the build system dealing with?
>>
>> Everything outside of "erlc" :)
>>
>> =A0- VPATH builds
>> =A0- Configuring the install to find the location of C libraries
>> =A0- Customising the install for users
>> =A0- Setting up the system infrastructure for CouchDB to function
>> =A0- Making sure things work across platforms
>> =A0- Building customised versions of binaries and scripts
>>
>> > I'm separating the
>> > config file (and similar files) problem and the icu and couch_js probl=
em.
>> I
>> > was hoping building those few C files wouldn't be bad, but I guess tha=
t
>> is
>> > not true from what you are saying?
>>
>> Doing the build system for an operating system daemon is Hard.
>>
>> > I'd still say it should be autotools, or whatever, inside an Erlang bu=
ild
>> > system.
>>
>> Basically, I would like to see the new build system separate the package
>> into two components. The CouchDB OTP application, which is build using s=
ome
>> Erlang appropriate build system (like rebar, or whatever) and everything
>> else. Everything else is handled by Autotools, like it is now. Autotools=
 is
>> also responsible for delegating the Erlang work off to the other build
>> system.
>>
>> A good way forward would be:
>>
>> =A0 =A0 =A0 =A0- Decide the minimum set of files needed for the Erlang b=
uild.
>>
>> =A0 =A0 =A0 =A0- Sandbox them into a directory along with the build file=
s.
>>
>> =A0 =A0 =A0 =A0- Create a Makefile.am file in that directory.
>>
>> =A0 =A0 =A0 =A0- Hook the new system into the Autotools system.
>>
>> That should work.
>>
>> We get to keep teh AWSUM POWAH of Autotools, and have an OTP app. :)
>>
>> Valediction,
>>
>> N
>>
>>
>

Tristan,

Apologies for not replying earlier. I meant to but got distracted by
something shiny.

As Noah points out there are quite a few issues that are going to be
pain points with transitioning towards a non Autotools build. I'm
confident that we can and will but there are a few things to consider
as we do this. The following is going to be a brain dump to try and
outline the issues as best as I can. These are in no specific order
but hopefully give a good outline of what we're facing.

I will also preference this with the fact that our current source tree
is in desperate need of a makeover. We've been talking about it for
awhile and its at the point that it just needs to happen. The rest
that follows is my gathering of thoughts for the move.

1. We have constraints. Noah's pointed out a few, but I don't even
think that list is exhaustive. A project like CouchDB needs to pay
attention to a lot of different details. I've said before that Noah is
the Chuck Norris of Autotools, so any proposed changes will need to be
cleared by him. He is the de facto BDFL for our build system and I
will vote with him lock step on any proposal.

2. There are lots of things in our current build system that may seem
innocuous at first, but turn out to be A Big Deal in some specific
circumstances that aren't always apparent at first glance.

3. A lot of changes I see in your CouchDB branch aren't going to fly.
Some reasons may seem trivial but this is me relying on knowledge from
Noah Chuck Norris Slater.

4. I'm not sure about Sinan, but rebar won't work out of the box for a
default build tool because of a lack of support for VPATH builds. At
this moment I'm still concentrating on making sure we can do a full
rebar build to support things like Erlang releases, but it's been
de-prioritized to a "secondary build system" status. If Sinan can do
VPATH builds, then I think it'd be a very good thing to support.

5. Erlang build knowledge is at some level, incompatible with
Autotools build knowledge. We need to find a common ground and figure
out where we can compromise to hopefully make both work. My current
shimmering of an idea for rebar was to have a make target that was
something like "make rebar" which would short circuit some parts of a
default build with rebar. Its possible that we can revamp part of the
build system to overcome the constraints of Autotools, but as I see it
the first move might just make a "developer friendly" alternate build
style.

6. Your CouchDB branch has some major source movement. Unfortunately
the ASF has not yet moved to having native git support. As such, any
such patches of this magnitude must be developed with Subversion in
mind. The reason is that we cannot develop this patch set in git and
apply it directly to svn because it would castrate our history
(because svn is file oriented). To address this I've started a small
one-off project where we can concentrate our changes so they are
reproducible. The code is up at [1], but its not been hashed out as
much as it needs to be. Last I hacked on it, I was getting ready to
add the ability to apply a series of patches after the required
barrage of svn commands.

7. Removing dependencies from the source tree is not going to happen
any time soon. I wish we didn't have to vendor so many projects, but
we have to remember that a majority of people building CouchDB are not
Erlangians. Forcing our community to install a number of Erlang
dependencies to build CouchDB would be a very large hurdle to
navigate. I know that there are projects like faxien and rebar's git
support to overcome this, but I don't feel that there is a solution
that sufficiently addresses this issue.

8. I don't know about Sinan or Rebar's platform support specifically.
Building C code portably is very very hard. I know that people detest
Autotools for such things, but its got an uncountable number of man
years addressing platform specific build issues. I don't think its
impossible to replace for building C code, I just see it as staring at
a mountain when people suggest it. IIRC, one of my first conversations
with Noah was switching to SCons. As much as it pains me now to admit,
I'm glad at he laughed at me then.

9. I'm quite happy to see someone familiar with Sinan and Erlang build
tools to jump into this conversation. The Erlangian part of the
CouchDB community has known that these sorts of changes are necessary,
but for the most part our frame of reference has been rebar. Seeing
another build system's requirement will help a *lot* in moving this
discussion forward.

10. Minor point, but our rough plan is to apply this level of massive
change just after we release 1.1.x so that the upheaval has time to be
used by the early adopters.

I suddenly can't think of anything else. I'm sure I'm forgetting
points that are important, but hopefully that starts to illustrate the
scope of what confronts this move. I think that its going to be a
difficult project and I'm extremely happy to see another Erlanger
involved in the discussion. My biggest fear with this move is that we
only manage to move to the Uncanny Valley of Erlang source trees. We
have rebar as a planned target. Sinan would be a great use case to
have so we can be more sure that our changes are enabling more Erlang
happiness.

Hopefully that wasn't too much of a brain dump to scare anyone. This
move definitely needs to happen and right now its mostly about making
sure we have our bases covered.

Paul Davis

[1] http://github.com/davisp/couchdb-srcmv