couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henri van den Bulk <>
Subject Re: Reconciling Data
Date Wed, 02 Nov 2011 21:07:28 GMT
Jens - 

Unfortunately the source system does not keep track of changes so there is no way of knowing
what the changes were. We've toyed around to maybe putting the data in an intermediate database
from which we can then do sql queries like not in and hash compares. However, this seems to
defeat the purpose of having couch.

Would it be possible to somehow post the ids to couch, like with filters using the request,
and then compare them?



On Nov 2, 2011, at 1:49 PM, Jens Alfke wrote:

> On Nov 2, 2011, at 9:20 AM, Henri van den Bulk wrote:
>> I'm in the process of writing an external program in Java that reconciles data in
CouchDB to a source system. One of the basic parts is to determine what data needs to be removed
from CouchDB. The good thing is that the Ids in CouchDB are the same as the Ids in the source
system. However, some initial test seem that the process is very slow in determining what
needs to be removed.
> It’s inevitably going to be inefficient to have to fetch _all_ the doc IDs from CouchDB
and _all_ the IDs from the other system and then scan them looking for differences.
> Instead, is there a way you can query the other system to find out which docs have been
removed since the last time you synced?
> —Jens

View raw message