From user-return-16133-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Thu May 5 23:00:14 2011 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1475D28E8 for ; Thu, 5 May 2011 23:00:14 +0000 (UTC) Received: (qmail 82764 invoked by uid 500); 5 May 2011 23:00:11 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 82674 invoked by uid 500); 5 May 2011 23:00:11 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 82666 invoked by uid 99); 5 May 2011 23:00:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 23:00:11 +0000 X-ASF-Spam-Status: No, hits=3.5 required=5.0 tests=FS_REPLICA,HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [128.18.84.113] (HELO mailgate-internal3.sri.com) (128.18.84.113) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 05 May 2011 23:00:03 +0000 Received: from brightmail-internal1.sri.com (128.18.84.121) by mailgate-internal3.sri.com with SMTP; 5 May 2011 22:59:40 -0000 X-AuditID: 80125479-b7c61ae000000cd9-14-4dc32bdc5e3b Received: from mars.esd.sri.com (mars.esd.sri.com [128.18.26.200]) by brightmail-internal1.sri.com (Symantec Brightmail Gateway) with SMTP id 86.18.03289.CDB23CD4; Thu, 5 May 2011 15:59:40 -0700 (PDT) MIME-version: 1.0 Received: from [192.12.16.187] by mars.esd.sri.com (Sun Java(tm) System Messaging Server 6.3-8.05 (built Sep 1 2009; 64bit)) with ESMTPSA id <0LKQ00GTBVV91130@mars.esd.sri.com> for user@couchdb.apache.org; Thu, 05 May 2011 15:59:34 -0700 (PDT) From: Jim Klo Content-type: multipart/signed; boundary=Apple-Mail-295--88334689; protocol="application/pkcs7-signature"; micalg=sha1 Subject: Re: Document Timestamp On Replication Date: Thu, 05 May 2011 15:59:39 -0700 In-reply-to: To: user@couchdb.apache.org References: <9946CE35511F1449ADF88A3F647DFE2816B0602907@HVXMSP8.us.lmco.com> <4DC067DD.5070508@facilityone.com> <9946CE35511F1449ADF88A3F647DFE2816B06F1724@HVXMSP8.us.lmco.com> <4DC163B2.10502@facilityone.com> <9946CE35511F1449ADF88A3F647DFE2816B06F1FA1@HVXMSP8.us.lmco.com> <4DC17B73.3010407@facilityone.com> <2C4218CE-6352-4F06-9F23-38E77CFFB254@sri.com> <4DC18FC7.10908@facilityone.com> <4DC1B552.9040603@facilityone.com> Message-id: <36976525-B616-41D3-AC3E-4A543242835C@sri.com> X-Mailer: Apple Mail (2.1084) X-Brightmail-Tracker: AAAAAA== X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-295--88334689 Content-Type: multipart/alternative; boundary=Apple-Mail-294--88334720 --Apple-Mail-294--88334720 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Besides our use not being applicable inside the browser, I'm not sure = that works for our use case, as we explicitly don't want any updates. = Our end solution ends up being a REST service to be used by others. We literally want a snapshot of the view at a specific point in time and = be able to operate on that snapshot without it changing. =20 A good physical analogy is I want to take a bucketful of water = (documents) from a lake (couchdb) being fed by a river (continous data = inserts/updates), and empty the bucket by the spoonful (pagination). = When I'm finished emptying the bucket, I'll go back to lake and refill = the bucket (new request), the lake now has more water (documents) since = I first started with the first bucketful since it's fed by the river = (continuous inserts/updates of new documents). In this scenario, the = state of the bucket is only effected by operations done with the spoon, = the lake has no effect upon the bucket. This is very close to what we = want to be able to achieve using CouchDB. To extend the analogy a bit to show what we do not want to occur, and = are desire to prevent [ as I believe this is close to how CouchDB = actually works]: if we had a hose that would continuously fill the = bucket from the lake while we are emptying the bucket by spoonful. We = could easily get into a state where the bucket begins to overflow or we = can never actually empty the bucket unless the process of emptying with = the spoon doesn't exceed the rate at which the bucket fills. I can't = change the rate at which bucket emptying occurs, nor can a predict or = change the rate the hose fills the bucket from the lake. Ultimately in = this extension - how do we get rid of the hose? It seems = update_seq=3Dtrue get's me part of the way there. What seems to be = missing and really what I need/want is a before=3D instead of a = since=3D. Jim Klo Senior Software Engineer Center for Software Engineering SRI International On May 5, 2011, at 2:07 PM, Chris Anderson wrote: > Not sure I follow all the requirements, but here is what I've done in = the past. >=20 > on page load: query the view with update_seq=3Dtrue >=20 > render the screen with up to date data as of seq X >=20 > open a changes request with since=3DX&include_docs=3Dtrue >=20 > each doc that comes down the pipe, run the map function again (in the > browser) and take whatever is emitted and stick it in your > datastructure that represents the view (or just directly update the > dom). also if an old version of the doc emitted something different, > remove whatever stuff in your in-page representation corresponds to > the old version of the doc. >=20 > now you have a screen that is kept up to date with a consistent > representation of what you'd get in a hard-reload, with a > transactional guarantee that no updates will be skipped. >=20 > Chris >=20 > On Wed, May 4, 2011 at 1:21 PM, Owen Marshall = wrote: >> On 05/04/2011 04:13 PM, Eli Stevens (Gmail) wrote: >>> 11:59 - Document D inserted on Node 2. Replication hasn't happened = yet. >>> 12:00 - First access of view page 1 on Node 1. Only A, B, C are = present. >>> 12:01 - D is replicated to Node 1. >>=20 >> Mmm, yes, you're absolutely correct; depending on that view would = carry >> with it the risk of an update race. It would (likely) work if = replicates >> were consistently low-latency, but that's not a guarantee. >>=20 >> Correct me if I'm wrong, but that view would work if: >>=20 >> 1. you capture last_seq from _changes pre-view run >> 2. run the view, capturing the output >> 3. check _changes for any updates since=3Dyour captured last_seq >> 4. filter those IDs out of your captured view. >>=20 >> Yuck. >>=20 >> -- >> Owen Marshall >> FacilityONE >> omarshall@facilityone.com | (502) 805-2126 >>=20 >>=20 >=20 >=20 >=20 > --=20 > Chris Anderson > http://jchrisa.net > http://couchbase.com --Apple-Mail-294--88334720 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Jim Klo
Senior Software Engineer
Center for Software Engineering
SRI = International




On May 5, 2011, at 2:07 PM, Chris Anderson wrote:

Not = sure I follow all the requirements, but here is what I've done in the = past.

on page load: query the view with = update_seq=3Dtrue

render the screen with up to date data as of = seq X

open a changes request with = since=3DX&include_docs=3Dtrue

each doc that comes down the = pipe, run the map function again (in the
browser) and take whatever = is emitted and stick it in your
datastructure that represents the = view (or just directly update the
dom). also if an old version of the = doc emitted something different,
remove whatever stuff in your = in-page representation corresponds to
the old version of the = doc.

now you have a screen that is kept up to date with a = consistent
representation of what you'd get in a hard-reload, with = a
transactional guarantee that no updates will be = skipped.

Chris

On Wed, May 4, 2011 at 1:21 PM, Owen = Marshall <omarshall@facilityone.com>= ; wrote:
On 05/04/2011 04:13 PM, Eli = Stevens (Gmail) wrote:
11:59 - Document D inserted on = Node 2.  Replication hasn't happened = yet.
12:00 - First access of view page 1 on Node 1.  Only = A, B, C are present.
12:01 - D is replicated to Node = 1.

Mmm, yes, = you're absolutely correct; depending on that view would = carry
with it the risk of an = update race. It would (likely) work if = replicates
were consistently = low-latency, but that's not a guarantee.

Correct me if = I'm wrong, but that view would work if:

1. you capture = last_seq from _changes pre-view run
2. run the view, capturing the = output
3. check _changes for = any updates since=3Dyour captured last_seq
4. filter those IDs out of your captured = view.

Yuck.

--
Owen = Marshall
FacilityONE
omarshall@facilityone.com = | (502) 805-2126





--
Chris Anderson
http://jchrisa.net
http://couchbase.com

= --Apple-Mail-294--88334720-- --Apple-Mail-295--88334689 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJ/zCCBO0w ggRWoAMCAQICEBZ7jcIF++u6rxPdCkJYyG0wDQYJKoZIhvcNAQEFBQAwgdgxCzAJBgNVBAYTAlVT MRowGAYDVQQKExFTUkkgSW50ZXJuYXRpb25hbDEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0 d29yazE7MDkGA1UECxMyVGVybXMgb2YgdXNlIGF0IGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9y cGEgKGMpMDIxMDAuBgNVBAsTJ0NsYXNzIDIgT25TaXRlIEluZGl2aWR1YWwgU3Vic2NyaWJlciBD QTEdMBsGA1UEAxMUU1JJIEludGVybmF0aW9uYWwgQ0EwHhcNMTEwMTE3MDAwMDAwWhcNMTIwMTE3 MjM1OTU5WjCBwjEaMBgGA1UEChQRU1JJIEludGVybmF0aW9uYWwxKDAmBgNVBAsUH0luZm9ybWF0 aW9uIFRlY2hub2xvZ3kgU2VydmljZXMxRjBEBgNVBAsTPXd3dy52ZXJpc2lnbi5jb20vcmVwb3Np dG9yeS9DUFMgSW5jb3JwLiBieSBSZWYuLExJQUIuTFREKGMpOTkxEjAQBgNVBAMTCUphbWVzIEts bzEeMBwGCSqGSIb3DQEJARYPamltLmtsb0BzcmkuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEA7X0QQ3Ag7/cRBwEgKfDEaOgXLwvnLhzmgY1bon3wSEK/ezUhlPhw8X/O4krsRp9v GKHAS5Z29ix+6B+PHJI3aptqCfaCT3ffu6MWFIyAhNaFNdvRBy8MhsD5lvjRffA7oysddhLWJ9AV madJBXjf0Fl+qoS/q0MbjsZSrQHeizYcv91SxcsWovgM6XoY87v0o7tHzUBWEr6jEOrvz50XKB8m pytqWAR8zLkp0NmsdgNk/PX6yXA3T4rPS690WOV3EDGK8pum2DIG7B319/lVeFQPdKdjNGpSivVt GXtT1W/KtpzW8Olmkn1sprupVOZXsKLU/MFXYPoFdR4pXoYViwIDAQABo4IBRjCCAUIwCQYDVR0T BAIwADCBrAYDVR0gBIGkMIGhMIGeBgtghkgBhvhFAQcXAjCBjjAoBggrBgEFBQcCARYcaHR0cHM6 Ly93d3cudmVyaXNpZ24uY29tL0NQUzBiBggrBgEFBQcCAjBWMBUWDlZlcmlTaWduLCBJbmMuMAMC AQEaPVZlcmlTaWduJ3MgQ1BTIGluY29ycC4gYnkgcmVmZXJlbmNlIGxpYWIuIGx0ZC4gKGMpOTcg VmVyaVNpZ24wCwYDVR0PBAQDAgWgMBEGCWCGSAGG+EIBAQQEAwIHgDBmBgNVHR8EXzBdMFugWaBX hlVodHRwOi8vb25zaXRlY3JsLnZlcmlzaWduLmNvbS9TUklJbnRlcm5hdGlvbmFsSW5mb3JtYXRp b25UZWNobm9sb2d5U2VydmljZXMvTGF0ZXN0Q1JMMA0GCSqGSIb3DQEBBQUAA4GBACf3MlYS4ssw EUnHTKP+v6xeJSPicFWwgYzS0iBOsuCpgUTTOSxPSPBwFNxY/plPMikXkK6rTGiIQUFXK59uqPV+ /1xXFpqvvt9/c0CqQDr8EfbbycaFyN8FaXQNV0gaqXDr/m4X2GZovm85T3osCKWzIijQzmr9xrQK 5yjpnTt3MIIFCjCCBHOgAwIBAgIQdRD9LNvKRXBSboyDbAKnbDANBgkqhkiG9w0BAQUFADCBwTEL MAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTwwOgYDVQQLEzNDbGFzcyAyIFB1 YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gRzIxOjA4BgNVBAsTMShjKSAx OTk4IFZlcmlTaWduLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxHzAdBgNVBAsTFlZl cmlTaWduIFRydXN0IE5ldHdvcmswHhcNMDIwOTIzMDAwMDAwWhcNMTIwOTIyMjM1OTU5WjCB2DEL MAkGA1UEBhMCVVMxGjAYBgNVBAoTEVNSSSBJbnRlcm5hdGlvbmFsMR8wHQYDVQQLExZWZXJpU2ln biBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2UgYXQgaHR0cHM6Ly93d3cudmVy aXNpZ24uY29tL3JwYSAoYykwMjEwMC4GA1UECxMnQ2xhc3MgMiBPblNpdGUgSW5kaXZpZHVhbCBT dWJzY3JpYmVyIENBMR0wGwYDVQQDExRTUkkgSW50ZXJuYXRpb25hbCBDQTCBnzANBgkqhkiG9w0B AQEFAAOBjQAwgYkCgYEAzvnUwmuZmBSSAFVb0qoC0hhUL1a6f+AIHw5UpxW5oRTjsDtUzsCa+6Yg GvKUlisrnI/tPZFrupvHVNQjRj05fhHiABFinwlnCA7J80x3gZlBMwHrgoKYribJ1GTVmc1R0FmA B4KYzBeZjJZiNpqLEsEb0ORdzJYb2/UZazjL/fkCAwEAAaOCAegwggHkMBIGA1UdEwEB/wQIMAYB Af8CAQAwRAYDVR0gBD0wOzA5BgtghkgBhvhFAQcXAjAqMCgGCCsGAQUFBwIBFhxodHRwczovL3d3 dy52ZXJpc2lnbi5jb20vcnBhMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwudmVyaXNpZ24u Y29tL3BjYTItZzIuY3JsMAsGA1UdDwQEAwIBBjARBglghkgBhvhCAQEEBAMCAQYwKAYDVR0RBCEw H6QdMBsxGTAXBgNVBAMTEFByaXZhdGVMYWJlbDItODIwHQYDVR0OBBYEFC1OfgnwbUVBEaxx2j87 9iZKf2RkMIHoBgNVHSMEgeAwgd2hgcekgcQwgcExCzAJBgNVBAYTAlVTMRcwFQYDVQQKEw5WZXJp U2lnbiwgSW5jLjE8MDoGA1UECxMzQ2xhc3MgMiBQdWJsaWMgUHJpbWFyeSBDZXJ0aWZpY2F0aW9u IEF1dGhvcml0eSAtIEcyMTowOAYDVQQLEzEoYykgMTk5OCBWZXJpU2lnbiwgSW5jLiAtIEZvciBh dXRob3JpemVkIHVzZSBvbmx5MR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrghEAuS9g zIifoXpGCbhbcGyKrzANBgkqhkiG9w0BAQUFAAOBgQAowFJw4GZ/4dbI1ncxPAvPGrV/aIB5Z8mZ e9tmn/CH+OcKSVI02h/Q5qbUD+P2hWMW3hBaQeCUG/YMWDgUXXEQKSeZYVGLpGdxkSAzV8VOQLIG JX3/1Lo4oo067Z8qZ0NLf6IH2SzZDEcDuFHGuc5Z0OM3Cghvwo6OX1oO37MiszGCBHswggR3AgEB MIHtMIHYMQswCQYDVQQGEwJVUzEaMBgGA1UEChMRU1JJIEludGVybmF0aW9uYWwxHzAdBgNVBAsT FlZlcmlTaWduIFRydXN0IE5ldHdvcmsxOzA5BgNVBAsTMlRlcm1zIG9mIHVzZSBhdCBodHRwczov L3d3dy52ZXJpc2lnbi5jb20vcnBhIChjKTAyMTAwLgYDVQQLEydDbGFzcyAyIE9uU2l0ZSBJbmRp dmlkdWFsIFN1YnNjcmliZXIgQ0ExHTAbBgNVBAMTFFNSSSBJbnRlcm5hdGlvbmFsIENBAhAWe43C Bfvruq8T3QpCWMhtMAkGBSsOAwIaBQCgggJiMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJ KoZIhvcNAQkFMQ8XDTExMDUwNTIyNTk0MFowIwYJKoZIhvcNAQkEMRYEFMTpaJl+fbt33trTYen8 n9I7GnW+MIH+BgkrBgEEAYI3EAQxgfAwge0wgdgxCzAJBgNVBAYTAlVTMRowGAYDVQQKExFTUkkg SW50ZXJuYXRpb25hbDEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazE7MDkGA1UECxMy VGVybXMgb2YgdXNlIGF0IGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9ycGEgKGMpMDIxMDAuBgNV BAsTJ0NsYXNzIDIgT25TaXRlIEluZGl2aWR1YWwgU3Vic2NyaWJlciBDQTEdMBsGA1UEAxMUU1JJ IEludGVybmF0aW9uYWwgQ0ECEBZ7jcIF++u6rxPdCkJYyG0wggEABgsqhkiG9w0BCRACCzGB8KCB 7TCB2DELMAkGA1UEBhMCVVMxGjAYBgNVBAoTEVNSSSBJbnRlcm5hdGlvbmFsMR8wHQYDVQQLExZW ZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2UgYXQgaHR0cHM6Ly93 d3cudmVyaXNpZ24uY29tL3JwYSAoYykwMjEwMC4GA1UECxMnQ2xhc3MgMiBPblNpdGUgSW5kaXZp ZHVhbCBTdWJzY3JpYmVyIENBMR0wGwYDVQQDExRTUkkgSW50ZXJuYXRpb25hbCBDQQIQFnuNwgX7 67qvE90KQljIbTANBgkqhkiG9w0BAQEFAASCAQAOCS55AGwLVONli+iQ9YzC5tyQe3EZ4EhtQdb+ 1FV2ZaFs0355xsCXgckTi0ub2PJXF1lbp6kpj5rl4GRsuIcicQNnbfsAgvBgJT0AuIV80xczmqav zs03FJtwnO/PaDSVKJvw38BI6yFZsa282mpisDqH4fvKtH9UQQy5m08FKT/hOqsJL6EiAgjRoWyz 8PXkYnSvIIQVLs8gtr440WpbXerJy+ZkXy/TER5PpSlM/m27QXMgh1ARmszmzrViaNIi4aaeUusf m7XACTdgwBUvsnd3oGvrnDxAoNkrocm9ffic+mq2xkuFqZ928QrU5ZiUaWUMwJum2+pAdQU4VxFa AAAAAAAA --Apple-Mail-295--88334689--