Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 07311929B for ; Wed, 2 Nov 2011 21:08:12 +0000 (UTC) Received: (qmail 84957 invoked by uid 500); 2 Nov 2011 21:08:10 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 84918 invoked by uid 500); 2 Nov 2011 21:08:10 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 84910 invoked by uid 99); 2 Nov 2011 21:08:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Nov 2011 21:08:10 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of hvandenbulk@gmail.com designates 209.85.215.180 as permitted sender) Received: from [209.85.215.180] (HELO mail-ey0-f180.google.com) (209.85.215.180) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Nov 2011 21:08:01 +0000 Received: by mail-ey0-f180.google.com with SMTP id 5so741136eyg.11 for ; Wed, 02 Nov 2011 14:07:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer; bh=O3xGEohgKLgzyLB69hMD66/WM4fGghlwrdqJm5CaA4A=; b=AKIFf3spZfpqP/2ceoPlTOZhZavsclpjXOytzEMb6SA30AsRd7/mnicbty8zY798KO pRNDtTQwQWkEc7G9E1EUedhFejz/EFTwZHEjWIQS0I4hPmHPdcTRrtIaoOJm+YUkn5G2 yzET6u1T3l9xGPeLp7hTlg0bBRkAxM1N5g+YI= Received: by 10.14.9.29 with SMTP id 29mr580003ees.88.1320268061313; Wed, 02 Nov 2011 14:07:41 -0700 (PDT) Received: from actiemac-2.denverwater.org ([159.143.1.2]) by mx.google.com with ESMTPS id 54sm9711864eex.8.2011.11.02.14.07.37 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 02 Nov 2011 14:07:40 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Apple Message framework v1084) Subject: Re: Reconciling Data From: Henri van den Bulk In-Reply-To: <7F5F85A9-D5B6-4779-A523-ECD6E90CBDD0@couchbase.com> Date: Wed, 2 Nov 2011 15:07:28 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <7DF2AD6D-6811-4CF5-95A7-CD6BECC92C4D@gmail.com> References: <7F5F85A9-D5B6-4779-A523-ECD6E90CBDD0@couchbase.com> To: user@couchdb.apache.org X-Mailer: Apple Mail (2.1084) X-Virus-Checked: Checked by ClamAV on apache.org Jens -=20 Unfortunately the source system does not keep track of changes so there = is no way of knowing what the changes were. We've toyed around to maybe = putting the data in an intermediate database from which we can then do = sql queries like not in and hash compares. However, this seems to defeat = the purpose of having couch. Would it be possible to somehow post the ids to couch, like with filters = using the request, and then compare them? Thanks, Henri On Nov 2, 2011, at 1:49 PM, Jens Alfke wrote: >=20 > On Nov 2, 2011, at 9:20 AM, Henri van den Bulk wrote: >=20 >> I'm in the process of writing an external program in Java that = reconciles data in CouchDB to a source system. One of the basic parts is = to determine what data needs to be removed from CouchDB. The good thing = is that the Ids in CouchDB are the same as the Ids in the source system. = However, some initial test seem that the process is very slow in = determining what needs to be removed. >=20 > It=92s inevitably going to be inefficient to have to fetch _all_ the = doc IDs from CouchDB and _all_ the IDs from the other system and then = scan them looking for differences. >=20 > Instead, is there a way you can query the other system to find out = which docs have been removed since the last time you synced? >=20 > =97Jens