Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: error (nike.apache.org: local policy)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\))
Subject: Re: [VOTE] Merge BigCouch
From: Jan Lehnardt <jan@apache.org>
In-Reply-To: 
 <CABvT1DHi8Ouj6S-kyf+EeWRNHUfm+RVnyDbkz_XAvV9OV-OjEw@mail.gmail.com>
Date: Fri, 10 May 2013 17:29:33 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <A1890522-CCB5-4FE6-9084-C07A8C72B11F@apache.org>
References: 
 <CABvT1DHi8Ouj6S-kyf+EeWRNHUfm+RVnyDbkz_XAvV9OV-OjEw@mail.gmail.com>
To: dev@couchdb.apache.org

+1

Jan
--

On May 7, 2013, at 21:34 , Robert Newson <rnewson@apache.org> wrote:

> Hi All,
>=20
> I propose to merge in the following work,
> https://github.com/rnewson/couchdb/tree/nebraska-merge-candidate to
> the official Apache CouchDB repository to a new branch (i.e, *not*
> master). Once there, the full CouchDB developer community can begin
> the work to incorporate the code here into an official release.
>=20
> You do not need to respond if you are in agreement. If there is no
> response in 72 hours, I will assume lazy consensus. If we reach
> consensus, I will start the IP clearance process and then the merge.
>=20
> As most of you know, Paul Davis and I recently sequestered ourselves
> away from society (in a place called Nebraska) to make this merge
> happen. I want to clarify that this work is not the BigCouch code you
> can see on github.com/cloudant/bigcouch but the Cloudant platform from
> which BigCouch was made. This means it is bang up to date with all the
> bug fixes and feature enhancements we've made in the last eighteen
> months or more. With that clarification made, here are our notes about
> what we achieved, what it means to the project and what isn't yet
> done;
>=20
> Nebraska Merge Roundup
>=20
>=20
> Stats:
>=20
>=20
> 1402 - total new commits
>=20
> 312 - commits written during the merge (will be reduced substantially
> by squashing)
>=20
> 408 - number of files changed
>=20
> 21,897 - number of lines added
>=20
> 4,277 - number of lines removed
>=20
> A retrospective:
>=20
> Bob Newson and I have come to the end of our merge sprint on getting
> BigCouch merged into Apache CouchDB. Its been a productive ten days
> here in the midwest. I managed to get Bob out to a bowling alley and
> he managed to get me to a sushi restaurant. In between the cultural
> exchanges we=92ve also managed to get a significant amount of work =
done
> on the merging as well.
>=20
>=20
> The current status of the merge is that we=92ve managed to resolve the
> differences in the single node execution of CouchDB. Both the
> JavaScript and Erlang test suites run with only one failure in the
> Erlang test suite due to a (deliberately) missing constraint on the
> number of operating system processes. This should be a relatively
> straightforward fix but was not prioritized during our limited time to
> work on the larger issues.
>=20
>=20
> We merged a large number of performance and stability enhancements
> back into single node CouchDB as well as a number of pure bug fixes.
> The biggest highlight is a brand new compactor that is both faster and
> creates smaller and better organized post-compaction databases.
>=20
>=20
> The current status of the merge is that single node operations should
> be completely unaffected as demonstrated by the test suite passing. On
> the other hand we haven=92t yet finished getting the clustered code
> merged to use some of the new changes in single node CouchDB. The
> single most significant portion of this work involves updates to the
> internal cluster API for views to use the recently rewritten indexer
> APIs. This should be a relatively straightforward bit of work that
> we=92ll be finishing over the next few weeks.
>=20
>=20
> All in all the merge work done so far has been quite successful. We=92ve=

> met our primary goal of getting the code merged in a fashion that does
> not affect single node operation while providing a starting point for
> the larger community to start reviewing the more significant changes
> made. Given the size of the diff between the two code bases we never
> expected to have a fully working clustered solution after ten days of
> work but we have succeeded in providing a base of work that will allow
> us and new contributors to get up to speed quickly.
>=20
>=20
> This work, coupled with work by Dave Cottlehuber and Beno=EEt Chesneau
> on updating the build system and various other internal updates, will
> provide a solid foundation for work going forward. Its an exciting
> time for CouchDB and anyone interested should keep an eye on the next
> few releases as we ramp up work on various core aspects of the
> database.
>=20
>=20
> We=92ve had an exciting few days working to prepare the road for an
> exciting next twelve to eighteen months. We hope that everyone will
> feel as excited as we do about the next twelve to eighteen months for
> Apache CouchDB. It should be an exciting ride.
>=20
>=20
>=20
> Things we got done
>=20
>=20
> * Large update to the source tree layout for Erlang applications. Each
> application now has a src/appname/(c_src|ebin|priv|src) structure. The
> build system has been updated.
>=20
> * Renamed src/couchdb to src/couch to match the Erlang convention of
> the top directory name matching the Erlang application name.
>=20
> * Imported Cloudant Erlang applications for clustered CouchDB. These
> are imported with their history by using git subtree and merging the
> top level commit. These are not external deps, development will happen
> within the CouchDB tree. The imported apps are:
>=20
>=20
>   * config - A couch_config replacement (Behavior is mostly identical
> to couch_config except how we listen for configuration changes
> internally to allow for smooth hot code upgrade).
>=20
>   * twig - An rsyslog source replacement for couch_log.
>=20
>   * rexi - An RPC library. Replaces Erlang=92s built-in rex =
application
> to avoid costly safety measures in the interest of performance and
> throughput.
>=20
>   * mem3 - The =93Dynamo=94 part of BigCouch responsible for managing =
cluster state
>=20
>   * fabric - The internal cluster-aware CouachDB API
>=20
>   * ets_lru - A small library application that provides an LRU
> implementation using a couple ets tables.
>=20
>   * ddoc_cache - Caches design documents on each node for use in
> design handler functions. This uses an ets_lru cache with a very short
> TTL.
>=20
>   * chttpd - The cluster aware HTTP layer
>=20
>=20
> Each imported app also had its build system updated to use Autotools
> along with the necessary updates noted above for the new application
> layouts for existing CouchDB erlang apps.
>=20
>=20
> * Merged a large amount of updates and fixes to couch_replicator based
> on work done internally at Cloudant. Unfortunately due to an error
> when we created our internal clone we lost a bit of history in some of
> the initial merge and have a big commit that affects
> couch_replicator_manager mostly. There are a number of other commits
> related to couch_replicator that resolve the single node vs. clustered
> differences. Some noticeable couch_replicator features:
>=20
>=20
>   * Optionally disable checkpoints so that replication can work when
> a source is read only. This should only be used for smaller databases
> as each replication call has to scan the entire source database on
> each invocation.
>=20
>   * A new changes_pending field in the _active_tasks output
>=20
>   * A fix to the continuous replication to automatically reconnect to
> a continuous changes feed when it sees a last_seq value. This allows
> for the source to selectively recycle the HTTP connections used which
> can be quite useful for =93permanent=94 replications.
>=20
>   * A multitude of smaller bug fix and stability enhancements.
>=20
>=20
> Updates to single node couch:
>=20
>=20
> * We changed the by_seq tree to store a copy of the #full_doc_info{}
> record instead of the #doc_info{} record. This gives significant speed
> improvements for compaction and replication and generally anything
> that needs to walk the by_seq tree and access document bodies
> internally.
>=20
> * We rewrote the compactor to be significantly faster as well as
> provides significantly better compacted databases. The two main halves
> are to use a temp file and replace the use of btrees in the temp file.
> The temp file only contains a temporary copy of the document ids. At
> the end of a compaction run we then rebuild the by_id btree in the
> compaction file from this temp file. The reason this helps so much is
> that the compaction is based on the update_seq btree, which for most
> cases means that the id tree is updated in roughly random order which
> is very bad for our append only btrees. By using the tmp file we can
> stream it in order back into the compacted db file at the end of
> compacting, generating a minimum amount of garbage in the process. The
> other upgrade was to implement an external merge sort module
> (couch_emsort) that is used with this temporary file.
>=20
> * Reject updates to design docs that introduce updates that break
> compilation for source code. Currently we only check map and reduce
> calls as the other should provide user visible errors instead of
> inexplicably empty views.
>=20
> because my OCD kicked in and I was unable to resist.
>=20
> * Reverted a change made a long time ago that uses two file
> descriptors for each database. See the todo list.
>=20
> * The reason to remove the second fd is so that we can rewrite ref
> counting. Better ref counting makes everyone happy, but the real
> reason is for this next bullet point:
>=20
> * Optimize couch_server to not require a round trip message pass for
> opening a database that=92s in the LRU. This is a significant
> performance boost for high concurrency access. We also optimized
> couch_server internals to not blow up when it=92s under load.
>=20
> * Introduce a #leaf{} record into the revision trees. This is never
> written to disk but makes internal code a lot cleaner when dealing
> with multiple versions of rev tree values.
>=20
> * Some changes to couch_changes to enable clustered access. Also some
> general cleanup
>=20
> * Internal changes to how CouchDB is booted in Erlang land. Not very
> sexy but this removes a lot of complicated un-Erlangy bits. We still
> have a bit of work left here.
>=20
> * btree chunk sizes are now configurable which can allow people to
> adjust the RAM/speed tradeoffs a bit more.
>=20
> * We now load update validation functions on the first write. This is
> a cluster-motivated change because the clustered version of this call
> is expensive and can lead to race conditions when opening a bunch of
> db shards simultaneously. This should be invisible to external
> clients.
>=20
> * Disabled conflict detection for local docs. They don=92t replicate =
so
> there=92s no point. This just led to clusters getting stuck and =
confused
> when there were lots of replications happening.
>=20
> * Changes to the multipart/mime parsing code. Necessary for clustered
> attachment uploads to split the incoming data  stream into N copies.
>=20
> * Don=92t use init:restart/0 when reloading the ICU driver. I think
> this has a bug. But we should rewrite this driver to be a NIF anyway.
>=20
> * New couch OS process manager. Significantly faster access to OS
> processes under heavy load. This replaces the hard limit with a soft
> limit. Process spawned over the soft limit will be used until they=92ve
> sat idle for a few minutes and then be closed. We have a todo item to
> add the hard ceiling back in (while keeping the soft ceiling).
>=20
> * Automatically replace some easily identifiable JS reductions with
> their builtin counterparts. Uses a regex to do the detection so its
> not too smart.
>=20
> * Improved view updater write batch.
>=20
> * Updates to couchjs=92 views.js to improve index update speeds
>=20
> * Updates to the _stats bultin reduce to allow reduces to work over
> emitted stats objects. Sometimes clients have summary data in a doc,
> and this allows them to combine stats if they follow the same pattern
> as the builtin expects.
>=20
> * Added a config:reload() that is accessible by POST=92ing to
> _config/_reload. Used by the JS tests to reset the config to what's on
> disk. This should prevent those test run failures where a test fails
> leaving the config in a bad state causing all subsequent tests to
> fail. I think. Maybe.
>=20
> * Databases are deleted synchronously in the test suite. We may need
> to address this on Windows. But it does seem to reduce the number of
> =93{error, file_exists}=94 failures.
>=20
> * I reimplemented the JS restartServer() function. There=92s a new
> _restart/token URL that will given a unique value for each instance of
> the Erlang VM. To run a restart we grab the current token value, hit
> _restart, then wait till we get a successful response with a different
> token. This appears to have made the restart strategy more robust.
>=20
>=20
>=20
> Things that need doing
>=20
>=20
> IP Clearance -
>=20
>=20
> We=92ll need to track down if we have the CCLA as well as look at each
> source file added to make sure each one is strictly from Cloudant or
> has an amenable license. I=92m pretty sure that the only one of =
interest
> is trunc_io.erl but we need to be thorough.
>=20
> documentation -
>=20
>=20
> There shouldn=92t be much here since the entire point of this merge =
was
> to not change the visible behavior of single node couch. A few things
> to add about the testing endpoints. Maybe an update to the compaction
> section mention the two new file names used.
>=20
>=20
> Copyright notices -
>=20
>=20
> We need to strip out copyright notices from individual files and make
> sure all files have a standard Apache License v2 header.
>=20
>=20
> clustered vhosts -
>=20
>=20
> We=92ve never implemented this at Cloudant. We either need to write a
> cluster or go back and tell people to use HAProxy (or similar) for
> such things.
>=20
>=20
> twig -
>=20
>=20
> We need to add another output type to twig that is configurable in
> some manner. Right now we spit out entire rsyslog records which isn=92t
> useful for most people. We=92ll need to implement the file writer from
> couch_log as well as update the _log HTTP handler to know when it can
> and can=92t expect to find data on disk.
>=20
>=20
> fabric -
>=20
>=20
> This is going to need a lot of work. Specifically view access is going
> to need to be updated to work with couch_mrview and friends.
>=20
>=20
> Boot a dev cluster -
>=20
>=20
> Once we fix up the clustering code we=92ll need to write instructions
> and scripts for pulling up a dev cluster.
>=20
>=20
> OTP stuff -
>=20
>=20
> We=92ve updated each app but we still need to pull some parts out of
> couchdb into their own application. Specifically the HTTP layer needs
> its own app. We could probably pull out the os process/query_servers
> as well as the os daemons and friends. Once done we need to update the
> supervision trees so we don=92t have things like couch starting and
> managing the replication manager process.
>=20
>=20
> ddoc_cache -
>=20
>=20
> Wire this up in couch_httpd_db to actually be used. Right now its only
> used in chttpd.
>=20
>=20
> couch_file upgrade -
>=20
>=20
> The revert to remove the second updater_fd from each #db{} record
> means that we=92re back in the original position of files appearing to
> slow down significantly under load. Since the initial hammer approach
> of just adding a second fd we=92ve since discovered that the =
underlying
> bug is due to the way that message passing works combined with
> Erlang=92s file io. Significantly though is the fact that the fix is
> rather simple to implement. A first draft of this work is on an old
> branch of mine here:
>=20
>=20
>   https://github.com/davisp/couchdb/commit/d856878
>=20
>=20
> finish the size calculating changes -
>=20
>=20
> The #leaf{} record change is to enable us to add more data size
> calculations. CouchDB master calculates a data size that account for
> all bytes that are active in a .couch file. Cloudant is interested in
> the total size of uncompressed docs and attachments minus the internal
> overhead of btrees. And there=92s a fourth number to calculate based =
on
> the compression level used. Having each of these numbers will be
> useful as well as the calculations they=92ll enable (ie, dead bytes in
> file, bytes used for overhead, compression ratio achieved, etc).
>=20
>=20
> couch_proc_manager -
>=20
>=20
> We need to implement the hard ceiling for capping the number of OS
> processes. We=92ve started seeing a need for this at Cloudant with =
some
> work loads so motivation to fix this is high. The only failing etap is
> the assertion of this ceiling.
>=20
>=20
> Synchronous db delete on Windows -
>=20
>=20
> I did this because running the test suite was driving me bonkers. I
> need to ask Dave about how this behaves on Windows (my guess is not
> well) but I think we can close things up so that it works better than
> the status quo.