From couchdb-user-return-774-apmail-incubator-couchdb-user-archive=incubator.apache.org@incubator.apache.org Tue Jul 15 23:52:22 2008 Return-Path: Delivered-To: apmail-incubator-couchdb-user-archive@locus.apache.org Received: (qmail 54933 invoked from network); 15 Jul 2008 23:52:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 15 Jul 2008 23:52:22 -0000 Received: (qmail 41794 invoked by uid 500); 15 Jul 2008 23:52:21 -0000 Delivered-To: apmail-incubator-couchdb-user-archive@incubator.apache.org Received: (qmail 41754 invoked by uid 500); 15 Jul 2008 23:52:21 -0000 Mailing-List: contact couchdb-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: couchdb-user@incubator.apache.org Delivered-To: mailing list couchdb-user@incubator.apache.org Received: (qmail 41741 invoked by uid 99); 15 Jul 2008 23:52:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jul 2008 16:52:21 -0700 X-ASF-Spam-Status: No, hits=1.4 required=10.0 tests=FS_REPLICA,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.68.5.9] (HELO relay00.pair.com) (209.68.5.9) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 15 Jul 2008 23:51:26 +0000 Received: (qmail 10814 invoked from network); 15 Jul 2008 23:50:48 -0000 Received: from unknown (HELO ?192.168.1.199?) (unknown) by unknown with SMTP; 15 Jul 2008 23:50:48 -0000 X-pair-Authenticated: 96.33.90.152 Message-Id: <6E29F30A-62C7-40FE-BEA2-BE5EFB89DF4C@apache.org> From: Damien Katz To: couchdb-user@incubator.apache.org In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v926) Subject: Re: replication usage? creating dupes? Date: Tue, 15 Jul 2008 19:50:48 -0400 References: <8949D592-56FE-42E4-94E5-388B681C2BB2@gmail.com> <6f7ea56f0807151040l3d1be21cvdac9913cd214f0c7@mail.gmail.com> X-Mailer: Apple Mail (2.926) X-Virus-Checked: Checked by ClamAV on apache.org Looks like the cause of the duplicates is a problem in the HTTP layer. It seems POST requests can fail to respond despite success, and then are retried by the HTTP client. I'm not sure if this is an HTTP client or a TCP layer thing that's doing the re-transmission. There seems to be a bug in Mochiweb that causes this to happen frequently when used by Safari, where it suddenly breaks the connection to the client. I'm not sure why it does this. However, upon investing this it appears this is always a potential problem, POSTing documents isn't idempotent, so a success followed by a network failure and retry can cause dupes. Therefore, don't use POST to create documents, generate the document ID yourself and use PUT. This will prevent duplicate documents from ever happening due to network failure. But it should not be the case that replication causes duplicates. Replication uses only idempotent requests, making that unlikely (but I won't say impossible). -Damien On Jul 15, 2008, at 3:24 PM, Chris Anderson wrote: > On Tue, Jul 15, 2008 at 10:40 AM, John Evans wrote: >> FWIW I've been running into these exact same issues. > > I'll chime in with a me too. I've worked around it by writing a > client-side replicator, that loads up all docs from both dbs, and > copies any missing docs over to the target db. My requirements mean I > don't need to worry about multiple versions of a doc, so this has > worked for me for the time being. It would be more convenient to have > fast, reliable replication. > > I'll commit the replicator to CouchRest this week. > > -- > Chris Anderson > http://jchris.mfdz.com