From user-return-21184-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Mon Jun 11 18:03:56 2012 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8DE86C9CE for ; Mon, 11 Jun 2012 18:03:56 +0000 (UTC) Received: (qmail 15952 invoked by uid 500); 11 Jun 2012 18:03:55 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 15919 invoked by uid 500); 11 Jun 2012 18:03:55 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 15908 invoked by uid 99); 11 Jun 2012 18:03:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jun 2012 18:03:55 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dave@muse.net.nz designates 209.85.220.180 as permitted sender) Received: from [209.85.220.180] (HELO mail-vc0-f180.google.com) (209.85.220.180) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jun 2012 18:03:48 +0000 Received: by vcbfk26 with SMTP id fk26so2684378vcb.11 for ; Mon, 11 Jun 2012 11:03:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=muse.net.nz; s=google; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=AYbZagf+Ledm4JZiih4qaIMFumcGSbbg8+svBLo3pJk=; b=CIxAWbwQv/tqxLzxJ1hubiowjkTZbZwA386y24zVR7r5JNRjpFPgvVxgttxcCTtbqR RUklvigeCfEjkVyeE2rYD8+ZXtYKLcSRWcjPJcYs3lMhavgsWA53lU+4448FugiUAEpa IRwovJXeDT0RfAYmobFIutW9AFj8Mb4QgeCfo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding :x-gm-message-state; bh=AYbZagf+Ledm4JZiih4qaIMFumcGSbbg8+svBLo3pJk=; b=OaEj1TLQEiQwHO+cQcTl0rG9y41lDy1q+2sXsKuWItvwbDRdD2I4Cc8VBCP5HX5Mz8 sYgdwzqUdgop4joLsgAVN36StN8X68ULPxbeTla3sOBSYzxwSMonXRdfgcgf/jbpG7lx 2QOuUQVOBx6YsiEcYYmeXMM4pMu7EUbVRfRYfF+LL+GTvr0RupZe2FxUydkGfiV1rCa9 fZZaRW/FWHMARklOsgPlnXSTkegf2alUYOxwEcboM47TeZPbcqyqtKHRVuxMF7pLJCJN 5pd0sixfe4pkq6ZUUFoBLZGHueZSmYX0RPLhxISdXZKn1KIOCjCKKYkCXNErp4q7PBYM Jo9w== MIME-Version: 1.0 Received: by 10.52.30.110 with SMTP id r14mr11154336vdh.0.1339437807624; Mon, 11 Jun 2012 11:03:27 -0700 (PDT) Received: by 10.52.33.172 with HTTP; Mon, 11 Jun 2012 11:03:27 -0700 (PDT) X-Originating-IP: [84.112.19.176] In-Reply-To: References: Date: Mon, 11 Jun 2012 20:03:27 +0200 Message-ID: Subject: Re: How to split the data over a period of time. From: Dave Cottlehuber To: user@couchdb.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Gm-Message-State: ALoCoQm91+8TuWV6DVIZsy+gZ7vNLJDKg2nhzgRMEgozScdByXS1736SAKA0J4nSsMDpa610u09k On 11 June 2012 17:42, Harikrishnan R wrote: > Hi Dave, > > =A0 Many thanks for your quick response. > > =A0 I am not updating any documents, I am keep on appending docs to datab= ase > with a specified timestamp. OK, so its a continuing series of new docs going into the DB. > =A0 One of my requirements is *unique login counts* between two specified > dates. > > =A0 my map function will emits like > > =A0 *{[2012, 6, 4], acc_id_1}* > * =A0 {[2012, 6, 4], acc_id_2}* > * =A0 {[2012, 6, 4], acc_id_3}* > * =A0 {[2012, 6, 5], acc_id_1}* > * =A0 {[2012, 6, 6], acc_id_4}* > * =A0 .....* > * =A0 ....* > > =A0 By using start key and end key I am able to get the unique counts. Are you aware you can use a reduce function and have couchdb manage that for you? > =A0 Here the problem is when the dates range specified may fall into back= up > or in-between. I don't follow. Are you using separate DBs per month, or some rotation sche= me? If you're not removing or aggregating data, then all the data will be in th= e DB. > =A0 1) How do I load backup db/views on-demand and merge the results. This is a bit of a hand-wavey question so I'll try my best. If you need to merge the views you'll need to merge the documents, (e.g. replicate backup -> main DB, then regenerate). Alternatively make your application aware and simply restore DB, regenerate views if you can't afford the downtime, then have the app work out how to consolidate the results. If you clarify how you're managing your data a better answer will be forthcoming. > =A0 2) Do I need a cronjob for taking the backup of views and db I'd not backup views *unless* you cannot afford the build time in case of a disaster *and* a replicated copy is not sufficient. But yes, cron is fine for triggering a backup. CoW - best explanation (of a memory model) http://hackerboss.com/copy-on-write-101-part-1-what-is-it/ and http://en.wikipedia.org/wiki/Snapshot_%28computer_storage%29 the disk-based version. Basically your filesystem stores multiple images of your filesystems, transparently directing reads of unmodified blocks to the originals, and sending writes to a new block. Thus the 90% of unchanged data doesn't take up any more space. From a backup point of view, this typically means that the backup to disk is super fast (less direct application downtime for DBs that don't allow direct copy like CouchDB) and on advanced/expensive disk arrays you can export that filesystem to another host and back it up indirectly, without impacting the load of the original volume. A+ Dave