From user-return-17025-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Thu Jul 7 10:24:47 2011 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7BBA145D7 for ; Thu, 7 Jul 2011 10:24:47 +0000 (UTC) Received: (qmail 12688 invoked by uid 500); 7 Jul 2011 10:24:45 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 12406 invoked by uid 500); 7 Jul 2011 10:24:31 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 12388 invoked by uid 99); 7 Jul 2011 10:24:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jul 2011 10:24:25 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of matt.goodall@gmail.com designates 209.85.216.180 as permitted sender) Received: from [209.85.216.180] (HELO mail-qy0-f180.google.com) (209.85.216.180) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jul 2011 10:24:17 +0000 Received: by qyk30 with SMTP id 30so554544qyk.11 for ; Thu, 07 Jul 2011 03:23:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Pk4t68n9x14T7S+cncbVo+RsZoUHpFpzqMDQQHZN4Eg=; b=pj3e1tcBsUTzolDKKGZw8P461pw3JC7F5RAsN9OJArM5hShg/1cvyxZwqE08DwGBlJ WUgw2RiDkIFTNn5wJ53hjfZ6poNK2xR2ZlpVw7ACZQ1m62D51i28O78EvVrZJ0T2KM5w 02r+PLdqsr9E6QkdyORnZD0+JCgwGSt7bzrWQ= MIME-Version: 1.0 Received: by 10.229.221.131 with SMTP id ic3mr426456qcb.239.1310034236924; Thu, 07 Jul 2011 03:23:56 -0700 (PDT) Received: by 10.229.86.195 with HTTP; Thu, 7 Jul 2011 03:23:56 -0700 (PDT) In-Reply-To: References: <4E150382.2050605@gmail.com> Date: Thu, 7 Jul 2011 11:23:56 +0100 Message-ID: Subject: Re: Iterating all documents From: Matt Goodall To: user@couchdb.apache.org Content-Type: multipart/alternative; boundary=0016362841a455e6a704a7781d50 X-Virus-Checked: Checked by ClamAV on apache.org --0016362841a455e6a704a7781d50 Content-Type: text/plain; charset=UTF-8 On 7 July 2011 11:17, Robert Newson wrote: > It would be better to read through _changes than _all_docs, surely? :) > Yep, I nearly always use the _changes feed for upgrades. As long as you make each document upgrade idempotent then it doesn't matter if it fails (sometimes schema-less docs are a little more schema-less than you expect ;-)) or you need to restart it. - Matt > > B. > > On 7 July 2011 10:03, Dan Sheedy wrote: > > oh. > > > > Sorry for the thread high jack. Good haircut btw, no regrets. > > > > On Thu, Jul 7, 2011 at 5:28 PM, Max Ogden wrote: > > > >> you should reconsider the haircut > >> > >> On Thu, Jul 7, 2011 at 12:25 AM, Dan Sheedy > wrote: > >> > >> > im heading off. might be late dinner. I'm getting my haircut. should > be > >> > home > >> > 8ish. > >> > > >> > On Thu, Jul 7, 2011 at 10:53 AM, Patrick Barnes > >> wrote: > >> > > >> > > Alternatively, how about you page through _all_docs? > >> > > > >> > > 1. Query http://server:5984/dbname/_**all_docs?limit=100< > >> > http://server:5984/dbname/_all_docs?limit=100> > >> > > > >> > > 2. Process that set. Store the id of the last document. > >> > > > >> > > 3. Query > http://server:5984/dbname/_**all_docs?limit=100&startkey=(** > >> > > last_id)&skip=1< > >> > > http://server:5984/dbname/_all_docs?limit=100&startkey=(last_id)&skip=1 > >> >to > >> > get the next set. > >> > > > >> > > 4. Repeat 2 and 3 until the returned set is empty. > >> > > > >> > > If your batch processing has to be able to resume after being > >> terminated, > >> > > just store the last_id in a file between each set. > >> > > > >> > > If you documents come from a specific view, you can do that too, the > >> only > >> > > difference would be that 'startkey' needs to be the last record's > view > >> > key, > >> > > and you may also need a 'startkey_docid=(last_id)' parameter if the > >> keys > >> > are > >> > > not unique. > >> > > > >> > > -Patrick > >> > > > >> > > > >> > > On 7/07/2011 9:41 AM, Matthias Eck wrote: > >> > > > >> > >> Hello, > >> > >> > >> > >> I need to add a new field to all documents in my database. > >> > >> > >> > >> To have a better control I wanted to do this by batches and defined > 2 > >> > >> views: > >> > >> documents_without_newfield > >> > >> documents_with_newfield > >> > >> > >> > >> My idea was to just take the first 100 returned by the > >> > >> documents_without_newfield view, calculate the new field for all of > >> > >> them, save them and take the next 100 etc. > >> > >> > >> > >> As it turns out the views do not seem to be updated immediately, > which > >> > >> means that the view documents_without_newfield returns about 95 > >> > >> documents that actually already had the new field calculated in the > >> > >> previous step. > >> > >> > >> > >> Can I force the view to update immediately so I can iterate through > >> > >> the all documents? > >> > >> > >> > >> Thanks, > >> > >> Matthias > >> > >> > >> > >> > >> > > >> > > > --0016362841a455e6a704a7781d50--