Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.219.43 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAJNb-9phov6DpcV55A4LaWz7m77b83Fpzu9s+qbF_TJ8W6i66A@mail.gmail.com>
References: 
 <CAJNb-9r3KNWaPstym3A2PUL8=4M2wXsr7dSfQ6A3W2u8zQCotg@mail.gmail.com>
 <CAJ_m3YDCo+CvYU6Pb4L7S54rB9B50RbF4a7R6OQP-_qi=kb=Uw@mail.gmail.com>
 <CAJNb-9phov6DpcV55A4LaWz7m77b83Fpzu9s+qbF_TJ8W6i66A@mail.gmail.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Wed, 12 Feb 2014 17:34:22 -0600
Message-ID: 
 <CAJ_m3YCrsOE1xDrQYzwgxckAJ4dKQSVN2EsxPa4u+cfvxzr=5g@mail.gmail.com>
Subject: Re: rcouch merge status
To: "dev@couchdb.apache.org" <dev@couchdb.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Found out how I messed up the import of couch_mrview:

https://gist.github.com/davisp/8848265/revisions

Notice that I accidentally switched the subtree split path to
couch_index on the fourth split. That'll do it. I've fixed this now
and updated 1994-merge-rcouch in couchdb-couch-mrview.

I see you've also done some other work on couch_index and
couch_replicator at least. I'll go ahead and rexport these for you but
please switch to the multi repository setup soon so that I don't have
to keep doing this. Git gymnastics are tiring and easy to botch as my
emails from today have clearly demonstrated.

On Wed, Feb 12, 2014 at 1:52 PM, Benoit Chesneau <bchesneau@gmail.com> wrote:
> On Wed, Feb 12, 2014 at 8:22 PM, Paul Davis <paul.joseph.davis@gmail.com> wrote:
>> On Wed, Feb 12, 2014 at 7:11 AM, Benoit Chesneau <bchesneau@gmail.com> wrote:
>>> Hi all,
>>>
>>> Sorry for this late status. I have been busy with a cold and needed work
>>> on the code.
>>>
>>> Anyway here is a quick status of what  have been done since the last
>>> status:
>>>
>>> 1) The view changes has been merged. It includes some code from what
>>> have beem given last year to the Apache foundation + unitests and a
>>> small refactoring that was already in action in other places I manage.
>>>
>>> The view change feature include:
>>>
>>> - index all changes in an index by sequences and by key: it's possible
>>>   to get all the changes of an index from a sequence or all changes for
>>> a key or in a range since a a sequence. Removed key are also returned
>>> which is new compared to the current code in the rcouch core core. (this
>>> new feature can also be useful later for someone that would like to
>>> build a feature chaining map/reduces steps eventually).
>>> - api to listen on index changes. You can retrieve all changes at once,
>>>   listen until a change happen (~= longpoll), listen continuously on all
>>> changes. Unlike the current changes internal api, this api is designed
>>> so it can be reused by any Erlang plugin the erlang way.
>>> - api to retrieve changes of a db using a view. If the indexation of
>>>   changes is enabled in a design doc, it will be used in place of the
>>> current view filter. If the indexation is not enabled the old and non
>>> efficient view filter is used.
>>> - Add a facility to index the views in a continuous manner. While this
>>>   api is designed internally to be reused, it's only enabled for now ehn
>>> listening on _changes. or by enable it in the change listener described
>>> above. When enabled the view index is refreshed evey N secs (can be set
>>> in the config file) or after N updates in the database (can be set in
>>> the config file).
>>> - The replication has also been been modified so you can replicate using
>>>   a view index as a filte. (And yes you can filter the changes using a
>>> key which is can give you a feature similar to the channels in the
>>> couchbase sync gateway)
>>> - the views changes has now unitests.
>>>
>>> 2) couch_log has been improved and is now using lager [1]. With this change
>>> instead to use our own way to log file we are using lager which improve
>>> the logging and  will make couchdb more tolerant in the face of large or
>>> many log messages, it won't out of memory the node.
>>>
>>> Note: Lager can handle multiple backend but we are for now only handling
>>> the file and console backend in the ini file. Other backends can be
>>> configured using the app.config file. Also the code is slightly
>>> different from the current one in rcouch and imported from a private
>>> branch in use somewhere: it allows to update the log configuration not
>>> only in the app.config but also in the ini and handle the log level /
>>> module.
>>>
>>>
>>> What is missing from rcouch:
>>>
>>> - validate doc on read [2]
>>> -  Write only databases [3]
>>> - Filter fields in included doc when returned from a _changes [4]
>>> - optimisations [5]
>>>
>>> Optimisations may be wait the final merge imo. Since they are quite
>>> similar to the one in bigcouch (with small changes) maybe it would be
>>> easier to just wait. @davisp @rnewson thoughts?
>>>
>>
>> I'd prefer to delay adding new things to anything until after we've
>> completed the merge. Any work that isn't towards finishing the merge
>> will be effectively doubled as we have to account for that during the
>> merge stuff.
>
> Well these opts are currently in rcouch. but different from bigcouch
> while I achieving the same result. I think in that case that the code
> in bigcouch is in a better shape and not really need to finish the
> merge
>>
>> That is to say, commits to master (specifically, Erlang changes),
>> 1843-feature-bigcouch, or 1994-merge-rcouch should be strictly to
>> getting the merge finished. Obviously that's with the caveat that bug
>> fixes for the release process are pushed through.
>>
>
> this what is done modulo the bugs discovered using the unitests suite :)
>
>>> For the others features I will make them available in one commit until
>>> tomorrow.
>>>
>>> ToDo :
>>>
>>> - add new features
>>
>> Is this just a general "get back to doing new work" or was there
>> something you mentioned that I'm not connecting this to?
>
> This is the 3 items missing mentionnend above.
>>
>>> - make rcouch use the splitted repositories created by @davisp (thanks a
>>>   lot!) . It's actually blocked because of a missing change in
>>> couch_mrview (unitests are missing). I will have a look on it later in
>>> the day. Hopefully @davisp will be able to help me on that. For now my
>>> attemps to resurrect them while preserving the history were a failure.
>>>
>>
>> Yep. Intended to do this last night but ran out of steam. I'll take a
>> look right now and get back to you with what I find.
>>
>>>
>>> Voila,
>>>
>>> Hopefully this merge will be finished by the end of the week. I profit
>>> on this mail to announce that I will take a vacation of 15 days at the
>>> end of the week, so I will be not available except for urgent things.
>
> cool thanks!
>>>
>>>
>>> - benoit
>>>
>>>
>>> [1] https://github.com/basho/lager
>>> [2] https://github.com/refuge/rcouch/wiki/Validate-documents-on-read
>>> [3] https://github.com/refuge/rcouch/wiki/Write-only-databases
>>> [4] https://github.com/refuge/rcouch/wiki/Couch-changes#wiki-filter-fields-in-included-doc-when-returned-from-a-_changes
>>> [5] https://github.com/refuge/rcouch/wiki/Source-Code-Changes#wiki-changes-in-core