Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of mitja@kleider.name designates
 178.63.68.96 as permitted sender)
Subject: Re: Replication filter ignoring deleted docs or changed values
From: Mitja Kleider <mitja@kleider.name>
To: user@couchdb.apache.org
In-Reply-To: 
 <CAGnHtbJ5xe6xWeEZxShq73OdB97U0wYjY6sg_Yr367Sq3e8kTA@mail.gmail.com>
References: <1310249296.4810.9.camel@laumo>
	 <CAGnHtbJkx4UnxkUwX9ONzg+VNKKXVRAcGx52xneM3E3wCv=2+A@mail.gmail.com>
	 <1310286816.3214.10.camel@laumo>
	 <CAGnHtbJ5xe6xWeEZxShq73OdB97U0wYjY6sg_Yr367Sq3e8kTA@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Date: Wed, 13 Jul 2011 15:52:09 +0200
Message-ID: <1310565129.6135.6.camel@laumo>
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit

Zachary,

On Sun, 2011-07-10 at 13:35 -0500, Zachary Zolton wrote:
> Mitja,
> 
> Perhaps you could record in some other field that the doc with type
> "bar" used to be type "foo".
> 
> Then way you can filter the replication by docs where:
>   type == "foo" || type == "bar" && old_type == "foo"
> 
> This should then replicate docs that change their type from "foo" to "bar".
> 
> To give any better advice, I'd need to hear more concretely what
> you're trying to do here.

I am working with geospatial data from the OpenStreetMap project.

Each document has infinitely many tags, similar to type="restaurant"
which might change in each revision.

I would also like to filter the geometry, to keep it simple something
like [x,y] and return (left <= x) && (x <= right) &&  (bottom <= y) &&
(y <= top).

The position might also change. I am afraid that I can not keep the
entire history because of the high amount of data.

The only possible other solution I have come up with is not using
filtered replication for updating the replicated target database, but
instead do full replication and filter in validate_doc_update(). Does
that make sense?


Mitja


> On Sunday, July 10, 2011, Mitja Kleider <mitja@kleider.name> wrote:
> > On Sat, 2011-07-09 at 17:26 -0500, Zachary Zolton wrote:
> >> Could you change your filter to something like this?
> >>
> >> function(doc) {
> >>   return doc._deleted || doc.type == 'foo';
> >> }
> >>
> >> That way you replicate all deleted docs.
> >
> > That would work, thanks. I tried to avoid replicating millions of
> > deleted documents together with a few hundred documents passing the
> > filter.
> >
> > A solution might be this validate_doc_update function on the target
> > database:
> >
> > function(newDoc, oldDoc, userCtx) {
> >     if (newDoc._deleted === true && !oldDoc) {
> >         throw({forbidden: 'do not create deleted docs'});
> >     }
> > }
> >
> > This way all deleted documents will be transmitted, but at least not
> > written if there is no previous revision.
> >
> >
> > The problem for changed values remains: when the source database changes
> > type='foo' to type='bar', the target database keeps the old revision
> > with type='foo'.
> >
> >
> > Mitja
> >
> >