Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of bchesneau@gmail.com designates
 209.85.216.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <A9E1E777-CA2A-41CE-A9F6-48FDFC3BD1D8@apache.org>
References: 
 <CAJNb-9r3KNWaPstym3A2PUL8=4M2wXsr7dSfQ6A3W2u8zQCotg@mail.gmail.com>
 <A9E1E777-CA2A-41CE-A9F6-48FDFC3BD1D8@apache.org>
From: Benoit Chesneau <bchesneau@gmail.com>
Date: Wed, 12 Feb 2014 16:53:48 +0100
Message-ID: 
 <CAJNb-9pSAzXL2e8vRB_OJdjLFTgdd8BsBMziyZRG5FnVUDv_9w@mail.gmail.com>
Subject: Re: rcouch merge status
To: "dev@couchdb.apache.org" <dev@couchdb.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Feb 12, 2014 at 3:25 PM, Adam Kocoloski <kocolosk@apache.org> wrote:
> On Feb 12, 2014, at 8:11 AM, Benoit Chesneau <bchesneau@gmail.com> wrote:
>
>> Hi all,
>>
>> Sorry for this late status. I have been busy with a cold and needed work
>> on the code.
>>
>> Anyway here is a quick status of what  have been done since the last
>> status:
>>
>> 1) The view changes has been merged. It includes some code from what
>> have beem given last year to the Apache foundation + unitests and a
>> small refactoring that was already in action in other places I manage.
>>
>> The view change feature include:
>>
>> - index all changes in an index by sequences and by key: it's possible
>>  to get all the changes of an index from a sequence or all changes for
>> a key or in a range since a a sequence. Removed key are also returned
>> which is new compared to the current code in the rcouch core core. (this
>> new feature can also be useful later for someone that would like to
>> build a feature chaining map/reduces steps eventually).
>> - api to listen on index changes. You can retrieve all changes at once,
>>  listen until a change happen (~= longpoll), listen continuously on all
>> changes. Unlike the current changes internal api, this api is designed
>> so it can be reused by any Erlang plugin the erlang way.
>> - api to retrieve changes of a db using a view. If the indexation of
>>  changes is enabled in a design doc, it will be used in place of the
>> current view filter. If the indexation is not enabled the old and non
>> efficient view filter is used.
>> - Add a facility to index the views in a continuous manner. While this
>>  api is designed internally to be reused, it's only enabled for now ehn
>> listening on _changes. or by enable it in the change listener described
>> above. When enabled the view index is refreshed evey N secs (can be set
>> in the config file) or after N updates in the database (can be set in
>> the config file).
>> - The replication has also been been modified so you can replicate using
>>  a view index as a filte. (And yes you can filter the changes using a
>> key which is can give you a feature similar to the channels in the
>> couchbase sync gateway)
>> - the views changes has now unitests.
>>
>> 2) couch_log has been improved and is now using lager [1]. With this change
>> instead to use our own way to log file we are using lager which improve
>> the logging and  will make couchdb more tolerant in the face of large or
>> many log messages, it won't out of memory the node.
>>
>> Note: Lager can handle multiple backend but we are for now only handling
>> the file and console backend in the ini file. Other backends can be
>> configured using the app.config file. Also the code is slightly
>> different from the current one in rcouch and imported from a private
>> branch in use somewhere: it allows to update the log configuration not
>> only in the app.config but also in the ini and handle the log level /
>> module.
>>
>>
>> What is missing from rcouch:
>>
>> - validate doc on read [2]
>> -  Write only databases [3]
>> - Filter fields in included doc when returned from a _changes [4]
>> - optimisations [5]
>>
>> Optimisations may be wait the final merge imo. Since they are quite
>> similar to the one in bigcouch (with small changes) maybe it would be
>> easier to just wait. @davisp @rnewson thoughts?
>>
>> For the others features I will make them available in one commit until
>> tomorrow.
>>
>> ToDo :
>>
>> - add new features
>> - make rcouch use the splitted repositories created by @davisp (thanks a
>>  lot!) . It's actually blocked because of a missing change in
>> couch_mrview (unitests are missing). I will have a look on it later in
>> the day. Hopefully @davisp will be able to help me on that. For now my
>> attemps to resurrect them while preserving the history were a failure.
>>
>>
>> Voila,
>>
>> Hopefully this merge will be finished by the end of the week. I profit
>> on this mail to announce that I will take a vacation of 15 days at the
>> end of the week, so I will be not available except for urgent things.
>>
>>
>> - benoit
>>
>>
>> [1] https://github.com/basho/lager
>> [2] https://github.com/refuge/rcouch/wiki/Validate-documents-on-read
>> [3] https://github.com/refuge/rcouch/wiki/Write-only-databases
>> [4] https://github.com/refuge/rcouch/wiki/Couch-changes#wiki-filter-fields-in-included-doc-when-returned-from-a-_changes
>> [5] https://github.com/refuge/rcouch/wiki/Source-Code-Changes#wiki-changes-in-core
>
> Cool stuff Benoit.  Are you saying you plan to redo these two bits of work in the individual repos by the end of the week?  Regards,
>
> Adam


Yes it should be done by the end of the week. The 3 others features
are really small to add. The multi repo should be easy once i figured
how to have couch_mrview with the history...

- benoit