couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Shorin <kxe...@gmail.com>
Subject Re: [VOTE] Merge BigCouch
Date Wed, 08 May 2013 19:47:06 GMT
+1
--
,,,^..^,,,


On Wed, May 8, 2013 at 11:20 PM, Garren Smith <garren@apache.org> wrote:
> +1
>
> On 08 May 2013, at 7:04 PM, Joan Touzet <wohali@apache.org> wrote:
>
>> +1
>>
>> On Tue, May 07, 2013 at 04:02:09PM -0500, Paul Davis wrote:
>>> +1
>>>
>>> On Tue, May 7, 2013 at 3:52 PM, Russell Branca <chewbranca@gmail.com> wrote:
>>>> +1
>>>>
>>>>
>>>> Very excited to see this! Great work!
>>>>
>>>>
>>>> -Russell
>>>>
>>>>
>>>> On Tue, May 7, 2013 at 1:44 PM, Robert Newson <rnewson@apache.org>
wrote:
>>>>
>>>>> FYI: A zip of this work is available at
>>>>> http://people.apache.org/~rnewson/dist/nebraska-merge-candidate.zip
>>>>> made by 'git archive -o nebraska-merge-candidate.zip
>>>>> nebraska-merge-candidate'
>>>>>
>>>>> On 7 May 2013 21:34, Robert Newson <rnewson@apache.org> wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> I propose to merge in the following work,
>>>>>> https://github.com/rnewson/couchdb/tree/nebraska-merge-candidate
to
>>>>>> the official Apache CouchDB repository to a new branch (i.e, *not*
>>>>>> master). Once there, the full CouchDB developer community can begin
>>>>>> the work to incorporate the code here into an official release.
>>>>>>
>>>>>> You do not need to respond if you are in agreement. If there is no
>>>>>> response in 72 hours, I will assume lazy consensus. If we reach
>>>>>> consensus, I will start the IP clearance process and then the merge.
>>>>>>
>>>>>> As most of you know, Paul Davis and I recently sequestered ourselves
>>>>>> away from society (in a place called Nebraska) to make this merge
>>>>>> happen. I want to clarify that this work is not the BigCouch code
you
>>>>>> can see on github.com/cloudant/bigcouch but the Cloudant platform
from
>>>>>> which BigCouch was made. This means it is bang up to date with all
the
>>>>>> bug fixes and feature enhancements we've made in the last eighteen
>>>>>> months or more. With that clarification made, here are our notes
about
>>>>>> what we achieved, what it means to the project and what isn't yet
>>>>>> done;
>>>>>>
>>>>>> Nebraska Merge Roundup
>>>>>>
>>>>>>
>>>>>> Stats:
>>>>>>
>>>>>>
>>>>>> 1402 - total new commits
>>>>>>
>>>>>> 312 - commits written during the merge (will be reduced substantially
>>>>>> by squashing)
>>>>>>
>>>>>> 408 - number of files changed
>>>>>>
>>>>>> 21,897 - number of lines added
>>>>>>
>>>>>> 4,277 - number of lines removed
>>>>>>
>>>>>> A retrospective:
>>>>>>
>>>>>> Bob Newson and I have come to the end of our merge sprint on getting
>>>>>> BigCouch merged into Apache CouchDB. Its been a productive ten days
>>>>>> here in the midwest. I managed to get Bob out to a bowling alley
and
>>>>>> he managed to get me to a sushi restaurant. In between the cultural
>>>>>> exchanges we’ve also managed to get a significant amount of work
done
>>>>>> on the merging as well.
>>>>>>
>>>>>>
>>>>>> The current status of the merge is that we’ve managed to resolve
the
>>>>>> differences in the single node execution of CouchDB. Both the
>>>>>> JavaScript and Erlang test suites run with only one failure in the
>>>>>> Erlang test suite due to a (deliberately) missing constraint on the
>>>>>> number of operating system processes. This should be a relatively
>>>>>> straightforward fix but was not prioritized during our limited time
to
>>>>>> work on the larger issues.
>>>>>>
>>>>>>
>>>>>> We merged a large number of performance and stability enhancements
>>>>>> back into single node CouchDB as well as a number of pure bug fixes.
>>>>>> The biggest highlight is a brand new compactor that is both faster
and
>>>>>> creates smaller and better organized post-compaction databases.
>>>>>>
>>>>>>
>>>>>> The current status of the merge is that single node operations should
>>>>>> be completely unaffected as demonstrated by the test suite passing.
On
>>>>>> the other hand we haven’t yet finished getting the clustered code
>>>>>> merged to use some of the new changes in single node CouchDB. The
>>>>>> single most significant portion of this work involves updates to
the
>>>>>> internal cluster API for views to use the recently rewritten indexer
>>>>>> APIs. This should be a relatively straightforward bit of work that
>>>>>> we’ll be finishing over the next few weeks.
>>>>>>
>>>>>>
>>>>>> All in all the merge work done so far has been quite successful.
We’ve
>>>>>> met our primary goal of getting the code merged in a fashion that
does
>>>>>> not affect single node operation while providing a starting point
for
>>>>>> the larger community to start reviewing the more significant changes
>>>>>> made. Given the size of the diff between the two code bases we never
>>>>>> expected to have a fully working clustered solution after ten days
of
>>>>>> work but we have succeeded in providing a base of work that will
allow
>>>>>> us and new contributors to get up to speed quickly.
>>>>>>
>>>>>>
>>>>>> This work, coupled with work by Dave Cottlehuber and Benoît Chesneau
>>>>>> on updating the build system and various other internal updates,
will
>>>>>> provide a solid foundation for work going forward. Its an exciting
>>>>>> time for CouchDB and anyone interested should keep an eye on the
next
>>>>>> few releases as we ramp up work on various core aspects of the
>>>>>> database.
>>>>>>
>>>>>>
>>>>>> We’ve had an exciting few days working to prepare the road for
an
>>>>>> exciting next twelve to eighteen months. We hope that everyone will
>>>>>> feel as excited as we do about the next twelve to eighteen months
for
>>>>>> Apache CouchDB. It should be an exciting ride.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Things we got done
>>>>>>
>>>>>>
>>>>>> * Large update to the source tree layout for Erlang applications.
Each
>>>>>> application now has a src/appname/(c_src|ebin|priv|src) structure.
The
>>>>>> build system has been updated.
>>>>>>
>>>>>> * Renamed src/couchdb to src/couch to match the Erlang convention
of
>>>>>> the top directory name matching the Erlang application name.
>>>>>>
>>>>>> * Imported Cloudant Erlang applications for clustered CouchDB. These
>>>>>> are imported with their history by using git subtree and merging
the
>>>>>> top level commit. These are not external deps, development will happen
>>>>>> within the CouchDB tree. The imported apps are:
>>>>>>
>>>>>>
>>>>>>   * config - A couch_config replacement (Behavior is mostly identical
>>>>>> to couch_config except how we listen for configuration changes
>>>>>> internally to allow for smooth hot code upgrade).
>>>>>>
>>>>>>   * twig - An rsyslog source replacement for couch_log.
>>>>>>
>>>>>>   * rexi - An RPC library. Replaces Erlang’s built-in rex application
>>>>>> to avoid costly safety measures in the interest of performance and
>>>>>> throughput.
>>>>>>
>>>>>>   * mem3 - The “Dynamo” part of BigCouch responsible for managing
>>>>> cluster state
>>>>>>
>>>>>>   * fabric - The internal cluster-aware CouachDB API
>>>>>>
>>>>>>   * ets_lru - A small library application that provides an LRU
>>>>>> implementation using a couple ets tables.
>>>>>>
>>>>>>   * ddoc_cache - Caches design documents on each node for use in
>>>>>> design handler functions. This uses an ets_lru cache with a very
short
>>>>>> TTL.
>>>>>>
>>>>>>   * chttpd - The cluster aware HTTP layer
>>>>>>
>>>>>>
>>>>>> Each imported app also had its build system updated to use Autotools
>>>>>> along with the necessary updates noted above for the new application
>>>>>> layouts for existing CouchDB erlang apps.
>>>>>>
>>>>>>
>>>>>> * Merged a large amount of updates and fixes to couch_replicator
based
>>>>>> on work done internally at Cloudant. Unfortunately due to an error
>>>>>> when we created our internal clone we lost a bit of history in some
of
>>>>>> the initial merge and have a big commit that affects
>>>>>> couch_replicator_manager mostly. There are a number of other commits
>>>>>> related to couch_replicator that resolve the single node vs. clustered
>>>>>> differences. Some noticeable couch_replicator features:
>>>>>>
>>>>>>
>>>>>>   * Optionally disable checkpoints so that replication can work when
>>>>>> a source is read only. This should only be used for smaller databases
>>>>>> as each replication call has to scan the entire source database on
>>>>>> each invocation.
>>>>>>
>>>>>>   * A new changes_pending field in the _active_tasks output
>>>>>>
>>>>>>   * A fix to the continuous replication to automatically reconnect
to
>>>>>> a continuous changes feed when it sees a last_seq value. This allows
>>>>>> for the source to selectively recycle the HTTP connections used which
>>>>>> can be quite useful for “permanent” replications.
>>>>>>
>>>>>>   * A multitude of smaller bug fix and stability enhancements.
>>>>>>
>>>>>>
>>>>>> Updates to single node couch:
>>>>>>
>>>>>>
>>>>>> * We changed the by_seq tree to store a copy of the #full_doc_info{}
>>>>>> record instead of the #doc_info{} record. This gives significant
speed
>>>>>> improvements for compaction and replication and generally anything
>>>>>> that needs to walk the by_seq tree and access document bodies
>>>>>> internally.
>>>>>>
>>>>>> * We rewrote the compactor to be significantly faster as well as
>>>>>> provides significantly better compacted databases. The two main halves
>>>>>> are to use a temp file and replace the use of btrees in the temp
file.
>>>>>> The temp file only contains a temporary copy of the document ids.
At
>>>>>> the end of a compaction run we then rebuild the by_id btree in the
>>>>>> compaction file from this temp file. The reason this helps so much
is
>>>>>> that the compaction is based on the update_seq btree, which for most
>>>>>> cases means that the id tree is updated in roughly random order which
>>>>>> is very bad for our append only btrees. By using the tmp file we
can
>>>>>> stream it in order back into the compacted db file at the end of
>>>>>> compacting, generating a minimum amount of garbage in the process.
The
>>>>>> other upgrade was to implement an external merge sort module
>>>>>> (couch_emsort) that is used with this temporary file.
>>>>>>
>>>>>> * Reject updates to design docs that introduce updates that break
>>>>>> compilation for source code. Currently we only check map and reduce
>>>>>> calls as the other should provide user visible errors instead of
>>>>>> inexplicably empty views.
>>>>>>
>>>>>> because my OCD kicked in and I was unable to resist.
>>>>>>
>>>>>> * Reverted a change made a long time ago that uses two file
>>>>>> descriptors for each database. See the todo list.
>>>>>>
>>>>>> * The reason to remove the second fd is so that we can rewrite ref
>>>>>> counting. Better ref counting makes everyone happy, but the real
>>>>>> reason is for this next bullet point:
>>>>>>
>>>>>> * Optimize couch_server to not require a round trip message pass
for
>>>>>> opening a database that’s in the LRU. This is a significant
>>>>>> performance boost for high concurrency access. We also optimized
>>>>>> couch_server internals to not blow up when it’s under load.
>>>>>>
>>>>>> * Introduce a #leaf{} record into the revision trees. This is never
>>>>>> written to disk but makes internal code a lot cleaner when dealing
>>>>>> with multiple versions of rev tree values.
>>>>>>
>>>>>> * Some changes to couch_changes to enable clustered access. Also
some
>>>>>> general cleanup
>>>>>>
>>>>>> * Internal changes to how CouchDB is booted in Erlang land. Not very
>>>>>> sexy but this removes a lot of complicated un-Erlangy bits. We still
>>>>>> have a bit of work left here.
>>>>>>
>>>>>> * btree chunk sizes are now configurable which can allow people to
>>>>>> adjust the RAM/speed tradeoffs a bit more.
>>>>>>
>>>>>> * We now load update validation functions on the first write. This
is
>>>>>> a cluster-motivated change because the clustered version of this
call
>>>>>> is expensive and can lead to race conditions when opening a bunch
of
>>>>>> db shards simultaneously. This should be invisible to external
>>>>>> clients.
>>>>>>
>>>>>> * Disabled conflict detection for local docs. They don’t replicate
so
>>>>>> there’s no point. This just led to clusters getting stuck and confused
>>>>>> when there were lots of replications happening.
>>>>>>
>>>>>> * Changes to the multipart/mime parsing code. Necessary for clustered
>>>>>> attachment uploads to split the incoming data  stream into N copies.
>>>>>>
>>>>>> * Don’t use init:restart/0 when reloading the ICU driver. I think
>>>>>> this has a bug. But we should rewrite this driver to be a NIF anyway.
>>>>>>
>>>>>> * New couch OS process manager. Significantly faster access to OS
>>>>>> processes under heavy load. This replaces the hard limit with a soft
>>>>>> limit. Process spawned over the soft limit will be used until they’ve
>>>>>> sat idle for a few minutes and then be closed. We have a todo item
to
>>>>>> add the hard ceiling back in (while keeping the soft ceiling).
>>>>>>
>>>>>> * Automatically replace some easily identifiable JS reductions with
>>>>>> their builtin counterparts. Uses a regex to do the detection so its
>>>>>> not too smart.
>>>>>>
>>>>>> * Improved view updater write batch.
>>>>>>
>>>>>> * Updates to couchjs’ views.js to improve index update speeds
>>>>>>
>>>>>> * Updates to the _stats bultin reduce to allow reduces to work over
>>>>>> emitted stats objects. Sometimes clients have summary data in a doc,
>>>>>> and this allows them to combine stats if they follow the same pattern
>>>>>> as the builtin expects.
>>>>>>
>>>>>> * Added a config:reload() that is accessible by POST’ing to
>>>>>> _config/_reload. Used by the JS tests to reset the config to what's
on
>>>>>> disk. This should prevent those test run failures where a test fails
>>>>>> leaving the config in a bad state causing all subsequent tests to
>>>>>> fail. I think. Maybe.
>>>>>>
>>>>>> * Databases are deleted synchronously in the test suite. We may need
>>>>>> to address this on Windows. But it does seem to reduce the number
of
>>>>>> “{error, file_exists}” failures.
>>>>>>
>>>>>> * I reimplemented the JS restartServer() function. There’s a new
>>>>>> _restart/token URL that will given a unique value for each instance
of
>>>>>> the Erlang VM. To run a restart we grab the current token value,
hit
>>>>>> _restart, then wait till we get a successful response with a different
>>>>>> token. This appears to have made the restart strategy more robust.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Things that need doing
>>>>>>
>>>>>>
>>>>>> IP Clearance -
>>>>>>
>>>>>>
>>>>>> We’ll need to track down if we have the CCLA as well as look at
each
>>>>>> source file added to make sure each one is strictly from Cloudant
or
>>>>>> has an amenable license. I’m pretty sure that the only one of interest
>>>>>> is trunc_io.erl but we need to be thorough.
>>>>>>
>>>>>> documentation -
>>>>>>
>>>>>>
>>>>>> There shouldn’t be much here since the entire point of this merge
was
>>>>>> to not change the visible behavior of single node couch. A few things
>>>>>> to add about the testing endpoints. Maybe an update to the compaction
>>>>>> section mention the two new file names used.
>>>>>>
>>>>>>
>>>>>> Copyright notices -
>>>>>>
>>>>>>
>>>>>> We need to strip out copyright notices from individual files and
make
>>>>>> sure all files have a standard Apache License v2 header.
>>>>>>
>>>>>>
>>>>>> clustered vhosts -
>>>>>>
>>>>>>
>>>>>> We’ve never implemented this at Cloudant. We either need to write
a
>>>>>> cluster or go back and tell people to use HAProxy (or similar) for
>>>>>> such things.
>>>>>>
>>>>>>
>>>>>> twig -
>>>>>>
>>>>>>
>>>>>> We need to add another output type to twig that is configurable in
>>>>>> some manner. Right now we spit out entire rsyslog records which isn’t
>>>>>> useful for most people. We’ll need to implement the file writer
from
>>>>>> couch_log as well as update the _log HTTP handler to know when it
can
>>>>>> and can’t expect to find data on disk.
>>>>>>
>>>>>>
>>>>>> fabric -
>>>>>>
>>>>>>
>>>>>> This is going to need a lot of work. Specifically view access is
going
>>>>>> to need to be updated to work with couch_mrview and friends.
>>>>>>
>>>>>>
>>>>>> Boot a dev cluster -
>>>>>>
>>>>>>
>>>>>> Once we fix up the clustering code we’ll need to write instructions
>>>>>> and scripts for pulling up a dev cluster.
>>>>>>
>>>>>>
>>>>>> OTP stuff -
>>>>>>
>>>>>>
>>>>>> We’ve updated each app but we still need to pull some parts out
of
>>>>>> couchdb into their own application. Specifically the HTTP layer needs
>>>>>> its own app. We could probably pull out the os process/query_servers
>>>>>> as well as the os daemons and friends. Once done we need to update
the
>>>>>> supervision trees so we don’t have things like couch starting and
>>>>>> managing the replication manager process.
>>>>>>
>>>>>>
>>>>>> ddoc_cache -
>>>>>>
>>>>>>
>>>>>> Wire this up in couch_httpd_db to actually be used. Right now its
only
>>>>>> used in chttpd.
>>>>>>
>>>>>>
>>>>>> couch_file upgrade -
>>>>>>
>>>>>>
>>>>>> The revert to remove the second updater_fd from each #db{} record
>>>>>> means that we’re back in the original position of files appearing
to
>>>>>> slow down significantly under load. Since the initial hammer approach
>>>>>> of just adding a second fd we’ve since discovered that the underlying
>>>>>> bug is due to the way that message passing works combined with
>>>>>> Erlang’s file io. Significantly though is the fact that the fix
is
>>>>>> rather simple to implement. A first draft of this work is on an old
>>>>>> branch of mine here:
>>>>>>
>>>>>>
>>>>>>   https://github.com/davisp/couchdb/commit/d856878
>>>>>>
>>>>>>
>>>>>> finish the size calculating changes -
>>>>>>
>>>>>>
>>>>>> The #leaf{} record change is to enable us to add more data size
>>>>>> calculations. CouchDB master calculates a data size that account
for
>>>>>> all bytes that are active in a .couch file. Cloudant is interested
in
>>>>>> the total size of uncompressed docs and attachments minus the internal
>>>>>> overhead of btrees. And there’s a fourth number to calculate based
on
>>>>>> the compression level used. Having each of these numbers will be
>>>>>> useful as well as the calculations they’ll enable (ie, dead bytes
in
>>>>>> file, bytes used for overhead, compression ratio achieved, etc).
>>>>>>
>>>>>>
>>>>>> couch_proc_manager -
>>>>>>
>>>>>>
>>>>>> We need to implement the hard ceiling for capping the number of OS
>>>>>> processes. We’ve started seeing a need for this at Cloudant with
some
>>>>>> work loads so motivation to fix this is high. The only failing etap
is
>>>>>> the assertion of this ceiling.
>>>>>>
>>>>>>
>>>>>> Synchronous db delete on Windows -
>>>>>>
>>>>>>
>>>>>> I did this because running the test suite was driving me bonkers.
I
>>>>>> need to ask Dave about how this behaves on Windows (my guess is not
>>>>>> well) but I think we can close things up so that it works better
than
>>>>>> the status quo.
>>>>>
>>
>> --
>> Joan Touzet | joant@atypical.net | wohali everywhere else
>

Mime
View raw message