Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of adam.kocoloski@gmail.com
 designates 74.125.83.180 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1078)
Subject: Re: overview of code organization?
From: Adam Kocoloski <kocolosk@apache.org>
In-Reply-To: <4BD1AAB6.9010800@meetinghouse.net>
Date: Fri, 23 Apr 2010 10:46:58 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <A133E5BF-6875-47C3-9375-B911B4244F29@apache.org>
References: <4BD197F0.40101@meetinghouse.net>
 <416ED41D-1803-4DAD-91F5-7670E452E531@apache.org>
 <4BD1AAB6.9010800@meetinghouse.net>
To: dev@couchdb.apache.org

On Apr 23, 2010, at 10:12 AM, Miles Fidelman wrote:

> Adam,
>=20
> Adam Kocoloski wrote:
>> On Apr 23, 2010, at 8:52 AM, Miles Fidelman wrote:
>>  =20
>>> - notes on the replication process (step-by-step, what happens when =
replication is invoked - what code modules are involved and so forth), =
and/or,
>>>    =20
>> couch_rep_* modules handle replication.  How familiar are you with =
Erlang/OTP?  couch_rep_sup is a supervisor for all replications, each of =
which has a couch_rep gen_server and changes_feed, missing_revs, reader, =
and writer processes.  Each of those processes handles one part of the =
"conversation" on the slide I pointed out to you two days ago.  Data =
flows from changes_feed ->  missing_revs ->  reader ->  writer.
>>  =20
>=20
> Pretty familiar with Erlang at a conceptual/system level; starting to =
take the time to get fluent in programming.  Haven't done functional =
languages in a long time.
>=20
>>> - an overview of the code for someone new to the project - what =
lives in what modules, how they string together - anything that might =
shortcut having to read through every module and make sense of things =
from scratch
>>>=20
>>> Anything - handwritten notes, slides from a code walkthrough, that =
kind of thing.
>>>    =20
>> Hi Miles, not to sound critical, but I don't think such a broad =
request will get you very far.  If you have specific questions I'll be =
happy to answer them.
>>  =20
> With all do respect... lots of projects maintain documentation of =
internals, particularly efforts focused on platform technologies =
intended for long-term and broad-based application.  Certainly in the =
world of commercial software development it's the rare project that =
doesn't have documentation providing a high level view of a large =
software system -- it's pretty hard to either bring new team members on =
board, or to perform long-term maintenance of code.  Granted that it's a =
bit harder to maintain this level of documentation on open-source =
projects without steady funding, but I will point at some examples:
> - Linux Kernel Internals: somewhat old (2.4), but =
http://tldp.org/LDP/lki/index.html (I know there are updates)
> - Apache HTTPD: http://httpd.apache.org/docs/2.2/developer/
> - MongoDB, documentation of replication internals: =
http://www.mongodb.org/display/DOCS/Replication+Internals
> - or even http://wiki.github.com/erlang/otp/routemap-source-tree - =
providing a basic overview of Erlang's internals
>=20
>> Please, take a shot at reading the code for the part you're =
interested in.  If you come across something you don't understand, send =
an email or join #couchdb on IRC.  Many of the devs hang out there =
regularly and can walk you through the code.  Best,
>>  =20
>=20
> It doesn't seem that unreasonable to at least ask whether Couch has =
some similar documentation floating around - if only at the level of =
notes put together by an individual developer, or for discussion among =
developers.
>=20
> Couch is certainly aiming at long-term viability as a platform for =
broad-based use, and seems to be aiming at being a broad-based =
open-source effort.  To succeed over the long term, it will NEED to have =
a good set of developer-level documentation.  "Read the code" is not a a =
long-term solution.
>=20
> Re. replication, in specific, the the couch_rep_* modules do not =
contain much in the way of comments.
>=20
> Personally, I've been involved in a LOT of network protocol-related =
work (BBN, back to the ARPANET days).  I've yet to see any kind of =
protocol work where someone hasn't jotted down at least a sequence =
diagram and some kind of dataflow diagram showing how all the pieces fit =
together.  More common is a full-blown ASN.1 description, and eventually =
an RFC in full gory detail.
>=20
> It does not seem unreasonable to ask if someone has jotted down notes =
about the full set of steps executed, and code modules involved, when =
Couch receives a "POST /_replicate" transaction.
>=20
> At the very least, it sure would be helpful to have something like:
> http://httpd.apache.org/docs/2.2/developer/request.html, or
> http://www.apachetutor.org/dev/request
> to detail the sequence of events and code involved in request =
processing.
>=20
> If, in fact, that kind of information has never been put on "paper," =
and lives only in the source code and a few people's heads, that scares =
me a lot vis-a-vis committing to Couch as a platform for any kind of =
serious project.
>=20
> Miles Fidelman

Hi Miles, I wasn't calling your request unreasonable, and I wasn't =
vouching for reading the code as the optimal source of developer =
documentation.  But it is what we have right now when you want to learn =
about things at module-level granularity.

It terms of broader architectural overviews, you may find Ricky Ho's set =
of articles useful:

http://horicky.blogspot.com/2008/10/couchdb-implementation.html

Regards, Adam