couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: View Processing Pipeline Architecture
Date Wed, 30 Nov 2016 14:55:23 GMT

> On 30 Nov 2016, at 15:34, Alan Kash <crudbug@gmail.com> wrote:
> 
> Thanks Jan.
> 
> I am not sure whether this is in the feature pipeline.

Not sure what you mean?

> 
> Another related query, *Live/Push queries* on views, the idea is to have a
> pub/sub kind of model, where clients subscribe to queries / views.

We have _changes on databases, and you can use view map functions (and
Mango queries) as filters.

This isn’t 100% what you want, but it get get you half way there.

There has been some preliminary work on offering proper _changes on views
as well, but conceptually, this is still a bit half-baked. I’m sure we’ll
get there eventually.

Best
Jan
--



> 
> 
> 
> On Wed, Nov 30, 2016 at 4:53 AM, Jan Lehnardt <jan@apache.org> wrote:
> 
>> 
>>> On 30 Nov 2016, at 00:06, Alan Kash <crudbug@gmail.com> wrote:
>>> 
>>> Yes, from the link it mentions about stale views.
>>> 
>>> Will be interested to know the merits of db pool vs view pool first
>> model.
>>> Logically, the queries are coming from top, and the view updates are
>>> propagated bottom up.
>> 
>> The internal data structure benefits from bulk inserts vs single inserts.
>> 
>> I.e. it takes (accumulated) less CPU, RAM, Disk, to update the view index
>> with 10 docs in one bulk operation instead of 10 individual operations.
>> 
>>> On a similar note, do the view updates leverage
>>> parallel processing in multicore systems.
>> 
>> In CouchDB 1.x: no, this is single threaded, albeit spread over two
>> operating system processes, but using a blocking protocol, so together,
>> they wouldn’t exhaust a single CPU.
>> 
>> In CouchDB 2.x, if you set up multiple shards per cluster node, you
>> basically get a CPU per shard for view computation, effectively using
>> multicore systems optimally.
>> 
>> Best
>> Jan
>> --
>> 
>> 
>>> 
>>> 
>>> 
>>> On Tue, Nov 29, 2016 at 5:49 PM, Robert Keizer <robert@keizer.ca> wrote:
>>> 
>>>> After a document is added.
>>>> 
>>>> My understanding is that the insertion to the main document pool happens
>>>> at the start, with views being triggered to update after the document
>>>> exists in that index.
>>>> 
>>>> Someone with more experience may correct me - but I'm fairly certain
>>>> this is why views can be stale (
>>>> http://docs.couchdb.org/en/2.0.0/api/ddoc/views.html )
>>>> 
>>>> 
>>>> On 2016-11-29 4:47 PM, Alan Kash wrote:
>>>>> Hello,
>>>>> 
>>>>> CouchDB starter here.
>>>>> 
>>>>> How are the MapReduce views updated ? Before or after a new document
is
>>>>> added into store ?
>>>>> 
>>>>> Do all document insertions go through some view process pipeline, then
>>>>> finally to base document pool ?  I am thinking in document stream
>>>>> processing terms, where document ingress hits the *views processes*
>> first
>>>>> before finally ending up in the pool.
>>>>> 
>>>>> Thanks
>>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>> 
>> 

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/


Mime
View raw message