couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Newson <rnew...@apache.org>
Subject Re: IP clearance (Was: Re: [VOTE] Merge BigCouch)
Date Thu, 16 May 2013 12:39:16 GMT
nah, I'll do it straight.

Can I do this? The docs say Officer or Member.

On 16 May 2013 13:37, Noah Slater <nslater@apache.org> wrote:
> git help svn
>
>
> On 16 May 2013 13:13, Robert Newson <rnewson@apache.org> wrote:
>
>> Righto. Now to remember how subversion works...
>>
>> On 15 May 2013 17:09, Noah Slater <nslater@apache.org> wrote:
>> > Okay.
>> >
>> > Start here:
>> >
>> > http://incubator.apache.org/ip-clearance/
>> >
>> > Then make a copy of this file:
>> >
>> >
>> http://svn.apache.org/repos/asf/incubator/public/trunk/content/ip-clearance/ip-clearance-template.xml
>> >
>> > This file, when rendered to HTML will look like:
>> >
>> > http://incubator.apache.org/ip-clearance/ip-clearance-template.html
>> >
>> > In your local copy, cut everything from:
>> >
>> >       <pre>-----8-&lt;---- cut here -------8-&lt;---- cut here
>> > -------8-&lt;---- cut here-------8-&lt;----</pre>
>> >
>> > To:
>> >
>> >       <pre>-----8-&lt;---- cut here -------8-&lt;---- cut here
>> > -------8-&lt;---- cut here-------8-&lt;----</pre>
>> >
>> > Now, add your copy back to Subversion here:
>> >
>> >
>> http://svn.apache.org/repos/asf/incubator/public/trunk/content/ip-clearance/
>> >
>> > Call it "couchdb-bigcouch.xml".
>> >
>> > In a few minutes, this will appear here:
>> >
>> > http://incubator.apache.org/ip-clearance/couchdb-bigcouch.html
>> >
>> > Now, it should be a simple matter of going through the doc and completing
>> > the checkpoints/sections.
>> >
>> > Here are the two previous ones we've done:
>> >
>> > http://incubator.apache.org/ip-clearance/couchdb-docs.html
>> >
>> > http://incubator.apache.org/ip-clearance/couchdb-fauxton.html
>> >
>> > Let me know if you get stuck on any of the checkpoints.
>> >
>> > Once you're done, let me know, and I will use my member karma to push it
>> > through the Incubator.
>> >
>> > Benoit, you may as well start your rcouch stuff at the same time using
>> this
>> > instructions. Obviously, you should pick "couchdb-rcouch.xml" instead.
>> But
>> > other than that, it's the same process.
>> >
>> > On 15 May 2013 16:24, Noah Slater <nslater@apache.org> wrote:
>> >
>> >> I can help! :)
>> >>
>> >>
>> >> On 15 May 2013 16:23, Robert Newson <rnewson@apache.org> wrote:
>> >>
>> >>> :)
>> >>>
>> >>> Jan, I think you said you'd help start the IP clearance bit?
>> >>>
>> >>> On 15 May 2013 15:03, Noah Slater <nslater@apache.org> wrote:
>> >>> > PARTY TIME 🎉
>> >>> >
>> >>> >
>> >>> > On 15 May 2013 10:40, Robert Newson <rnewson@apache.org>
wrote:
>> >>> >
>> >>> >> Thanks everyone.
>> >>> >>
>> >>> >> The tally is;
>> >>> >>
>> >>> >> 13 +1's
>> >>> >>
>> >>> >> The vote passes. We'll now move on to IP clearance. Once that's
done
>> >>> >> the work will arrive on a feature branch in our main git repository.
>> >>> >>
>> >>> >> B.
>> >>> >>
>> >>> >>
>> >>> >> On 13 May 2013 04:31, Jason Smith <jhs@iriscouch.com>
wrote:
>> >>> >> > Sorry, just catching up.
>> >>> >> >
>> >>> >> > +1
>> >>> >> >
>> >>> >> > On Fri, May 10, 2013 at 4:29 PM, Jan Lehnardt <jan@apache.org>
>> >>> wrote:
>> >>> >> >> +1
>> >>> >> >>
>> >>> >> >> Jan
>> >>> >> >> --
>> >>> >> >>
>> >>> >> >> On May 7, 2013, at 21:34 , Robert Newson <rnewson@apache.org>
>> >>> wrote:
>> >>> >> >>
>> >>> >> >>> Hi All,
>> >>> >> >>>
>> >>> >> >>> I propose to merge in the following work,
>> >>> >> >>>
>> https://github.com/rnewson/couchdb/tree/nebraska-merge-candidateto
>> >>> >> >>> the official Apache CouchDB repository to a new
branch (i.e,
>> *not*
>> >>> >> >>> master). Once there, the full CouchDB developer
community can
>> begin
>> >>> >> >>> the work to incorporate the code here into an
official release.
>> >>> >> >>>
>> >>> >> >>> You do not need to respond if you are in agreement.
If there is
>> no
>> >>> >> >>> response in 72 hours, I will assume lazy consensus.
If we reach
>> >>> >> >>> consensus, I will start the IP clearance process
and then the
>> >>> merge.
>> >>> >> >>>
>> >>> >> >>> As most of you know, Paul Davis and I recently
sequestered
>> >>> ourselves
>> >>> >> >>> away from society (in a place called Nebraska)
to make this
>> merge
>> >>> >> >>> happen. I want to clarify that this work is not
the BigCouch
>> code
>> >>> you
>> >>> >> >>> can see on github.com/cloudant/bigcouch but the
Cloudant
>> platform
>> >>> from
>> >>> >> >>> which BigCouch was made. This means it is bang
up to date with
>> all
>> >>> the
>> >>> >> >>> bug fixes and feature enhancements we've made
in the last
>> eighteen
>> >>> >> >>> months or more. With that clarification made,
here are our notes
>> >>> about
>> >>> >> >>> what we achieved, what it means to the project
and what isn't
>> yet
>> >>> >> >>> done;
>> >>> >> >>>
>> >>> >> >>> Nebraska Merge Roundup
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Stats:
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> 1402 - total new commits
>> >>> >> >>>
>> >>> >> >>> 312 - commits written during the merge (will be
reduced
>> >>> substantially
>> >>> >> >>> by squashing)
>> >>> >> >>>
>> >>> >> >>> 408 - number of files changed
>> >>> >> >>>
>> >>> >> >>> 21,897 - number of lines added
>> >>> >> >>>
>> >>> >> >>> 4,277 - number of lines removed
>> >>> >> >>>
>> >>> >> >>> A retrospective:
>> >>> >> >>>
>> >>> >> >>> Bob Newson and I have come to the end of our merge
sprint on
>> >>> getting
>> >>> >> >>> BigCouch merged into Apache CouchDB. Its been
a productive ten
>> days
>> >>> >> >>> here in the midwest. I managed to get Bob out
to a bowling alley
>> >>> and
>> >>> >> >>> he managed to get me to a sushi restaurant. In
between the
>> cultural
>> >>> >> >>> exchanges we’ve also managed to get a significant
amount of work
>> >>> done
>> >>> >> >>> on the merging as well.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> The current status of the merge is that we’ve
managed to resolve
>> >>> the
>> >>> >> >>> differences in the single node execution of CouchDB.
Both the
>> >>> >> >>> JavaScript and Erlang test suites run with only
one failure in
>> the
>> >>> >> >>> Erlang test suite due to a (deliberately) missing
constraint on
>> the
>> >>> >> >>> number of operating system processes. This should
be a
>> relatively
>> >>> >> >>> straightforward fix but was not prioritized during
our limited
>> >>> time to
>> >>> >> >>> work on the larger issues.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> We merged a large number of performance and stability
>> enhancements
>> >>> >> >>> back into single node CouchDB as well as a number
of pure bug
>> >>> fixes.
>> >>> >> >>> The biggest highlight is a brand new compactor
that is both
>> faster
>> >>> and
>> >>> >> >>> creates smaller and better organized post-compaction
databases.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> The current status of the merge is that single
node operations
>> >>> should
>> >>> >> >>> be completely unaffected as demonstrated by the
test suite
>> >>> passing. On
>> >>> >> >>> the other hand we haven’t yet finished getting
the clustered
>> code
>> >>> >> >>> merged to use some of the new changes in single
node CouchDB.
>> The
>> >>> >> >>> single most significant portion of this work involves
updates to
>> >>> the
>> >>> >> >>> internal cluster API for views to use the recently
rewritten
>> >>> indexer
>> >>> >> >>> APIs. This should be a relatively straightforward
bit of work
>> that
>> >>> >> >>> we’ll be finishing over the next few weeks.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> All in all the merge work done so far has been
quite successful.
>> >>> We’ve
>> >>> >> >>> met our primary goal of getting the code merged
in a fashion
>> that
>> >>> does
>> >>> >> >>> not affect single node operation while providing
a starting
>> point
>> >>> for
>> >>> >> >>> the larger community to start reviewing the more
significant
>> >>> changes
>> >>> >> >>> made. Given the size of the diff between the two
code bases we
>> >>> never
>> >>> >> >>> expected to have a fully working clustered solution
after ten
>> days
>> >>> of
>> >>> >> >>> work but we have succeeded in providing a base
of work that will
>> >>> allow
>> >>> >> >>> us and new contributors to get up to speed quickly.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> This work, coupled with work by Dave Cottlehuber
and Benoît
>> >>> Chesneau
>> >>> >> >>> on updating the build system and various other
internal updates,
>> >>> will
>> >>> >> >>> provide a solid foundation for work going forward.
Its an
>> exciting
>> >>> >> >>> time for CouchDB and anyone interested should
keep an eye on the
>> >>> next
>> >>> >> >>> few releases as we ramp up work on various core
aspects of the
>> >>> >> >>> database.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> We’ve had an exciting few days working to prepare
the road for
>> an
>> >>> >> >>> exciting next twelve to eighteen months. We hope
that everyone
>> will
>> >>> >> >>> feel as excited as we do about the next twelve
to eighteen
>> months
>> >>> for
>> >>> >> >>> Apache CouchDB. It should be an exciting ride.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Things we got done
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> * Large update to the source tree layout for Erlang
>> applications.
>> >>> Each
>> >>> >> >>> application now has a src/appname/(c_src|ebin|priv|src)
>> structure.
>> >>> The
>> >>> >> >>> build system has been updated.
>> >>> >> >>>
>> >>> >> >>> * Renamed src/couchdb to src/couch to match the
Erlang
>> convention
>> >>> of
>> >>> >> >>> the top directory name matching the Erlang application
name.
>> >>> >> >>>
>> >>> >> >>> * Imported Cloudant Erlang applications for clustered
CouchDB.
>> >>> These
>> >>> >> >>> are imported with their history by using git subtree
and merging
>> >>> the
>> >>> >> >>> top level commit. These are not external deps,
development will
>> >>> happen
>> >>> >> >>> within the CouchDB tree. The imported apps are:
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>   * config - A couch_config replacement (Behavior
is mostly
>> >>> identical
>> >>> >> >>> to couch_config except how we listen for configuration
changes
>> >>> >> >>> internally to allow for smooth hot code upgrade).
>> >>> >> >>>
>> >>> >> >>>   * twig - An rsyslog source replacement for couch_log.
>> >>> >> >>>
>> >>> >> >>>   * rexi - An RPC library. Replaces Erlang’s
built-in rex
>> >>> application
>> >>> >> >>> to avoid costly safety measures in the interest
of performance
>> and
>> >>> >> >>> throughput.
>> >>> >> >>>
>> >>> >> >>>   * mem3 - The “Dynamo” part of BigCouch responsible
for
>> managing
>> >>> >> cluster state
>> >>> >> >>>
>> >>> >> >>>   * fabric - The internal cluster-aware CouachDB
API
>> >>> >> >>>
>> >>> >> >>>   * ets_lru - A small library application that
provides an LRU
>> >>> >> >>> implementation using a couple ets tables.
>> >>> >> >>>
>> >>> >> >>>   * ddoc_cache - Caches design documents on each
node for use in
>> >>> >> >>> design handler functions. This uses an ets_lru
cache with a very
>> >>> short
>> >>> >> >>> TTL.
>> >>> >> >>>
>> >>> >> >>>   * chttpd - The cluster aware HTTP layer
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Each imported app also had its build system updated
to use
>> >>> Autotools
>> >>> >> >>> along with the necessary updates noted above for
the new
>> >>> application
>> >>> >> >>> layouts for existing CouchDB erlang apps.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> * Merged a large amount of updates and fixes to
couch_replicator
>> >>> based
>> >>> >> >>> on work done internally at Cloudant. Unfortunately
due to an
>> error
>> >>> >> >>> when we created our internal clone we lost a bit
of history in
>> >>> some of
>> >>> >> >>> the initial merge and have a big commit that affects
>> >>> >> >>> couch_replicator_manager mostly. There are a number
of other
>> >>> commits
>> >>> >> >>> related to couch_replicator that resolve the single
node vs.
>> >>> clustered
>> >>> >> >>> differences. Some noticeable couch_replicator
features:
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>   * Optionally disable checkpoints so that replication
can work
>> >>> when
>> >>> >> >>> a source is read only. This should only be used
for smaller
>> >>> databases
>> >>> >> >>> as each replication call has to scan the entire
source database
>> on
>> >>> >> >>> each invocation.
>> >>> >> >>>
>> >>> >> >>>   * A new changes_pending field in the _active_tasks
output
>> >>> >> >>>
>> >>> >> >>>   * A fix to the continuous replication to automatically
>> reconnect
>> >>> to
>> >>> >> >>> a continuous changes feed when it sees a last_seq
value. This
>> >>> allows
>> >>> >> >>> for the source to selectively recycle the HTTP
connections used
>> >>> which
>> >>> >> >>> can be quite useful for “permanent” replications.
>> >>> >> >>>
>> >>> >> >>>   * A multitude of smaller bug fix and stability
enhancements.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Updates to single node couch:
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> * We changed the by_seq tree to store a copy of
the
>> >>> #full_doc_info{}
>> >>> >> >>> record instead of the #doc_info{} record. This
gives significant
>> >>> speed
>> >>> >> >>> improvements for compaction and replication and
generally
>> anything
>> >>> >> >>> that needs to walk the by_seq tree and access
document bodies
>> >>> >> >>> internally.
>> >>> >> >>>
>> >>> >> >>> * We rewrote the compactor to be significantly
faster as well as
>> >>> >> >>> provides significantly better compacted databases.
The two main
>> >>> halves
>> >>> >> >>> are to use a temp file and replace the use of
btrees in the temp
>> >>> file.
>> >>> >> >>> The temp file only contains a temporary copy of
the document
>> ids.
>> >>> At
>> >>> >> >>> the end of a compaction run we then rebuild the
by_id btree in
>> the
>> >>> >> >>> compaction file from this temp file. The reason
this helps so
>> much
>> >>> is
>> >>> >> >>> that the compaction is based on the update_seq
btree, which for
>> >>> most
>> >>> >> >>> cases means that the id tree is updated in roughly
random order
>> >>> which
>> >>> >> >>> is very bad for our append only btrees. By using
the tmp file we
>> >>> can
>> >>> >> >>> stream it in order back into the compacted db
file at the end of
>> >>> >> >>> compacting, generating a minimum amount of garbage
in the
>> process.
>> >>> The
>> >>> >> >>> other upgrade was to implement an external merge
sort module
>> >>> >> >>> (couch_emsort) that is used with this temporary
file.
>> >>> >> >>>
>> >>> >> >>> * Reject updates to design docs that introduce
updates that
>> break
>> >>> >> >>> compilation for source code. Currently we only
check map and
>> reduce
>> >>> >> >>> calls as the other should provide user visible
errors instead of
>> >>> >> >>> inexplicably empty views.
>> >>> >> >>>
>> >>> >> >>> because my OCD kicked in and I was unable to resist.
>> >>> >> >>>
>> >>> >> >>> * Reverted a change made a long time ago that
uses two file
>> >>> >> >>> descriptors for each database. See the todo list.
>> >>> >> >>>
>> >>> >> >>> * The reason to remove the second fd is so that
we can rewrite
>> ref
>> >>> >> >>> counting. Better ref counting makes everyone happy,
but the real
>> >>> >> >>> reason is for this next bullet point:
>> >>> >> >>>
>> >>> >> >>> * Optimize couch_server to not require a round
trip message pass
>> >>> for
>> >>> >> >>> opening a database that’s in the LRU. This is
a significant
>> >>> >> >>> performance boost for high concurrency access.
We also optimized
>> >>> >> >>> couch_server internals to not blow up when it’s
under load.
>> >>> >> >>>
>> >>> >> >>> * Introduce a #leaf{} record into the revision
trees. This is
>> never
>> >>> >> >>> written to disk but makes internal code a lot
cleaner when
>> dealing
>> >>> >> >>> with multiple versions of rev tree values.
>> >>> >> >>>
>> >>> >> >>> * Some changes to couch_changes to enable clustered
access. Also
>> >>> some
>> >>> >> >>> general cleanup
>> >>> >> >>>
>> >>> >> >>> * Internal changes to how CouchDB is booted in
Erlang land. Not
>> >>> very
>> >>> >> >>> sexy but this removes a lot of complicated un-Erlangy
bits. We
>> >>> still
>> >>> >> >>> have a bit of work left here.
>> >>> >> >>>
>> >>> >> >>> * btree chunk sizes are now configurable which
can allow people
>> to
>> >>> >> >>> adjust the RAM/speed tradeoffs a bit more.
>> >>> >> >>>
>> >>> >> >>> * We now load update validation functions on the
first write.
>> This
>> >>> is
>> >>> >> >>> a cluster-motivated change because the clustered
version of this
>> >>> call
>> >>> >> >>> is expensive and can lead to race conditions when
opening a
>> bunch
>> >>> of
>> >>> >> >>> db shards simultaneously. This should be invisible
to external
>> >>> >> >>> clients.
>> >>> >> >>>
>> >>> >> >>> * Disabled conflict detection for local docs.
They don’t
>> replicate
>> >>> so
>> >>> >> >>> there’s no point. This just led to clusters
getting stuck and
>> >>> confused
>> >>> >> >>> when there were lots of replications happening.
>> >>> >> >>>
>> >>> >> >>> * Changes to the multipart/mime parsing code.
Necessary for
>> >>> clustered
>> >>> >> >>> attachment uploads to split the incoming data
 stream into N
>> >>> copies.
>> >>> >> >>>
>> >>> >> >>> * Don’t use init:restart/0 when reloading the
ICU driver. I
>> think
>> >>> >> >>> this has a bug. But we should rewrite this driver
to be a NIF
>> >>> anyway.
>> >>> >> >>>
>> >>> >> >>> * New couch OS process manager. Significantly
faster access to
>> OS
>> >>> >> >>> processes under heavy load. This replaces the
hard limit with a
>> >>> soft
>> >>> >> >>> limit. Process spawned over the soft limit will
be used until
>> >>> they’ve
>> >>> >> >>> sat idle for a few minutes and then be closed.
We have a todo
>> item
>> >>> to
>> >>> >> >>> add the hard ceiling back in (while keeping the
soft ceiling).
>> >>> >> >>>
>> >>> >> >>> * Automatically replace some easily identifiable
JS reductions
>> with
>> >>> >> >>> their builtin counterparts. Uses a regex to do
the detection so
>> its
>> >>> >> >>> not too smart.
>> >>> >> >>>
>> >>> >> >>> * Improved view updater write batch.
>> >>> >> >>>
>> >>> >> >>> * Updates to couchjs’ views.js to improve index
update speeds
>> >>> >> >>>
>> >>> >> >>> * Updates to the _stats bultin reduce to allow
reduces to work
>> over
>> >>> >> >>> emitted stats objects. Sometimes clients have
summary data in a
>> >>> doc,
>> >>> >> >>> and this allows them to combine stats if they
follow the same
>> >>> pattern
>> >>> >> >>> as the builtin expects.
>> >>> >> >>>
>> >>> >> >>> * Added a config:reload() that is accessible by
POST’ing to
>> >>> >> >>> _config/_reload. Used by the JS tests to reset
the config to
>> >>> what's on
>> >>> >> >>> disk. This should prevent those test run failures
where a test
>> >>> fails
>> >>> >> >>> leaving the config in a bad state causing all
subsequent tests
>> to
>> >>> >> >>> fail. I think. Maybe.
>> >>> >> >>>
>> >>> >> >>> * Databases are deleted synchronously in the test
suite. We may
>> >>> need
>> >>> >> >>> to address this on Windows. But it does seem to
reduce the
>> number
>> >>> of
>> >>> >> >>> “{error, file_exists}” failures.
>> >>> >> >>>
>> >>> >> >>> * I reimplemented the JS restartServer() function.
There’s a new
>> >>> >> >>> _restart/token URL that will given a unique value
for each
>> >>> instance of
>> >>> >> >>> the Erlang VM. To run a restart we grab the current
token value,
>> >>> hit
>> >>> >> >>> _restart, then wait till we get a successful response
with a
>> >>> different
>> >>> >> >>> token. This appears to have made the restart strategy
more
>> robust.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Things that need doing
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> IP Clearance -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> We’ll need to track down if we have the CCLA
as well as look at
>> >>> each
>> >>> >> >>> source file added to make sure each one is strictly
from
>> Cloudant
>> >>> or
>> >>> >> >>> has an amenable license. I’m pretty sure that
the only one of
>> >>> interest
>> >>> >> >>> is trunc_io.erl but we need to be thorough.
>> >>> >> >>>
>> >>> >> >>> documentation -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> There shouldn’t be much here since the entire
point of this
>> merge
>> >>> was
>> >>> >> >>> to not change the visible behavior of single node
couch. A few
>> >>> things
>> >>> >> >>> to add about the testing endpoints. Maybe an update
to the
>> >>> compaction
>> >>> >> >>> section mention the two new file names used.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Copyright notices -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> We need to strip out copyright notices from individual
files and
>> >>> make
>> >>> >> >>> sure all files have a standard Apache License
v2 header.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> clustered vhosts -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> We’ve never implemented this at Cloudant. We
either need to
>> write a
>> >>> >> >>> cluster or go back and tell people to use HAProxy
(or similar)
>> for
>> >>> >> >>> such things.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> twig -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> We need to add another output type to twig that
is configurable
>> in
>> >>> >> >>> some manner. Right now we spit out entire rsyslog
records which
>> >>> isn’t
>> >>> >> >>> useful for most people. We’ll need to implement
the file writer
>> >>> from
>> >>> >> >>> couch_log as well as update the _log HTTP handler
to know when
>> it
>> >>> can
>> >>> >> >>> and can’t expect to find data on disk.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> fabric -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> This is going to need a lot of work. Specifically
view access is
>> >>> going
>> >>> >> >>> to need to be updated to work with couch_mrview
and friends.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Boot a dev cluster -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Once we fix up the clustering code we’ll need
to write
>> instructions
>> >>> >> >>> and scripts for pulling up a dev cluster.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> OTP stuff -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> We’ve updated each app but we still need to
pull some parts out
>> of
>> >>> >> >>> couchdb into their own application. Specifically
the HTTP layer
>> >>> needs
>> >>> >> >>> its own app. We could probably pull out the os
>> >>> process/query_servers
>> >>> >> >>> as well as the os daemons and friends. Once done
we need to
>> update
>> >>> the
>> >>> >> >>> supervision trees so we don’t have things like
couch starting
>> and
>> >>> >> >>> managing the replication manager process.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> ddoc_cache -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Wire this up in couch_httpd_db to actually be
used. Right now
>> its
>> >>> only
>> >>> >> >>> used in chttpd.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> couch_file upgrade -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> The revert to remove the second updater_fd from
each #db{}
>> record
>> >>> >> >>> means that we’re back in the original position
of files
>> appearing
>> >>> to
>> >>> >> >>> slow down significantly under load. Since the
initial hammer
>> >>> approach
>> >>> >> >>> of just adding a second fd we’ve since discovered
that the
>> >>> underlying
>> >>> >> >>> bug is due to the way that message passing works
combined with
>> >>> >> >>> Erlang’s file io. Significantly though is the
fact that the fix
>> is
>> >>> >> >>> rather simple to implement. A first draft of this
work is on an
>> old
>> >>> >> >>> branch of mine here:
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>   https://github.com/davisp/couchdb/commit/d856878
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> finish the size calculating changes -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> The #leaf{} record change is to enable us to add
more data size
>> >>> >> >>> calculations. CouchDB master calculates a data
size that account
>> >>> for
>> >>> >> >>> all bytes that are active in a .couch file. Cloudant
is
>> interested
>> >>> in
>> >>> >> >>> the total size of uncompressed docs and attachments
minus the
>> >>> internal
>> >>> >> >>> overhead of btrees. And there’s a fourth number
to calculate
>> based
>> >>> on
>> >>> >> >>> the compression level used. Having each of these
numbers will be
>> >>> >> >>> useful as well as the calculations they’ll enable
(ie, dead
>> bytes
>> >>> in
>> >>> >> >>> file, bytes used for overhead, compression ratio
achieved, etc).
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> couch_proc_manager -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> We need to implement the hard ceiling for capping
the number of
>> OS
>> >>> >> >>> processes. We’ve started seeing a need for this
at Cloudant with
>> >>> some
>> >>> >> >>> work loads so motivation to fix this is high.
The only failing
>> >>> etap is
>> >>> >> >>> the assertion of this ceiling.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> Synchronous db delete on Windows -
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> I did this because running the test suite was
driving me
>> bonkers. I
>> >>> >> >>> need to ask Dave about how this behaves on Windows
(my guess is
>> not
>> >>> >> >>> well) but I think we can close things up so that
it works better
>> >>> than
>> >>> >> >>> the status quo.
>> >>> >> >>
>> >>> >> >
>> >>> >> >
>> >>> >> >
>> >>> >> > --
>> >>> >> > Iris Couch
>> >>> >>
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > NS
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> NS
>> >>
>> >
>> >
>> >
>> > --
>> > NS
>>
>
>
>
> --
> NS

Mime
View raw message