From user-return-19959-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Mon Mar 5 16:24:33 2012 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9B60B9621 for ; Mon, 5 Mar 2012 16:24:33 +0000 (UTC) Received: (qmail 4575 invoked by uid 500); 5 Mar 2012 16:24:32 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 4542 invoked by uid 500); 5 Mar 2012 16:24:32 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 4533 invoked by uid 99); 5 Mar 2012 16:24:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Mar 2012 16:24:32 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of zachary.zolton@gmail.com designates 209.85.160.52 as permitted sender) Received: from [209.85.160.52] (HELO mail-pw0-f52.google.com) (209.85.160.52) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Mar 2012 16:24:25 +0000 Received: by pbcuo15 with SMTP id uo15so1978330pbc.11 for ; Mon, 05 Mar 2012 08:24:05 -0800 (PST) Received-SPF: pass (google.com: domain of zachary.zolton@gmail.com designates 10.68.232.162 as permitted sender) client-ip=10.68.232.162; Authentication-Results: mr.google.com; spf=pass (google.com: domain of zachary.zolton@gmail.com designates 10.68.232.162 as permitted sender) smtp.mail=zachary.zolton@gmail.com; dkim=pass header.i=zachary.zolton@gmail.com Received: from mr.google.com ([10.68.232.162]) by 10.68.232.162 with SMTP id tp2mr40902497pbc.165.1330964645529 (num_hops = 1); Mon, 05 Mar 2012 08:24:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=UPKlkLASuMEhuRB4YTWoND3+VfaxBVJbVjQXTfjy4MU=; b=Vuwhbr16ug3fAVkfHXLcKRin/96/Vc2K9FQXYK+WXFlG4D5FKC2bUvbDdzDArMSdHV xmdNqknjFrc3PN6YJ8VxxTyM2ljB9dFYxGf2JG5EHN/kcW4MV9BhxLU5kEKp22YNkXi3 X+KnG630BRLmMmSZDljrTNrAHjxkU/0knYbY9mTZftHx+iw0Jz4CSP20ynAfV4kzArK4 jVnEe9b1PyDUdAE+gK66KzwBMS0RD+YzR1Zk/ueEiHZFZd1968pC4PGo69RJpGmkuXkg judh7IoSWgX5kTLw8DP6wblWQ0EpULN7nWswKcMA2nFjBTRUHugXwPj1W0i9ru1Lf6jG Y8sw== Received: by 10.68.232.162 with SMTP id tp2mr35307253pbc.165.1330964645490; Mon, 05 Mar 2012 08:24:05 -0800 (PST) MIME-Version: 1.0 Received: by 10.68.195.106 with HTTP; Mon, 5 Mar 2012 08:23:35 -0800 (PST) From: Zachary Zolton Date: Mon, 5 Mar 2012 10:23:35 -0600 Message-ID: Subject: Strategy for reliable _changes feed workers To: user@couchdb.apache.org Content-Type: multipart/alternative; boundary=047d7b33d7b0e730a004ba815af0 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b33d7b0e730a004ba815af0 Content-Type: text/plain; charset=ISO-8859-1 Hi, I've enjoyed using _changes for non-critical functionality such as keeping a user interface up-to-date, but it also seems like a nice paradigm for background processing as well. For reliable background processing, my mind is inevitably drawn towards the following questions: * What is the last sequence number processed? * Have we already attempted to process this update? * How many times have we failed this update failed? Unfortunately, this all sounds like global state, which makes me feel un-relaxed. Now, you could help alleviate this by adding a work queue to your stack, but it feels like CouchDB could handle this style of processing without it. Also, someone program would have to then sit between the _changes feed and my background processors, which seems like a bottleneck. How are you using the _changes feed for reliable background processing? Cheers, Zach --047d7b33d7b0e730a004ba815af0--