From user-return-23331-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Thu Jan 24 02:33:03 2013 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 302EAE958 for ; Thu, 24 Jan 2013 02:33:03 +0000 (UTC) Received: (qmail 82453 invoked by uid 500); 24 Jan 2013 02:33:01 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 82346 invoked by uid 500); 24 Jan 2013 02:33:00 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 82337 invoked by uid 99); 24 Jan 2013 02:33:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jan 2013 02:33:00 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [128.18.84.133] (HELO brightmail-internal4.sri.com) (128.18.84.133) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jan 2013 02:32:53 +0000 X-AuditID: 80125485-b7fc16d00000612f-a5-51009d3fc28b Received: from exchange-hub01.SRI.COM (exchange-hub01.SRI.COM [128.18.23.153]) (using TLS with cipher AES128-SHA (AES128-SHA/128 bits)) (Client did not present a certificate) by brightmail-internal4.sri.com (SRI Internal SMTP Gateway) with SMTP id D1.27.24879.F3D90015; Wed, 23 Jan 2013 18:32:32 -0800 (PST) Received: from EXCHANGE-DB08.SRI.COM ([fe80::a11e:7c21:6886:9a20]) by exchange-hub01.SRI.COM ([2002:8012:1799::8012:1799]) with mapi id 14.02.0298.004; Wed, 23 Jan 2013 18:32:27 -0800 From: Jim Klo To: "" Subject: Re: Bulk insert-or-update Thread-Topic: Bulk insert-or-update Thread-Index: AQHN+PZPhEmvt9/x1UiK3alZV8w0LphYSkwA Date: Thu, 24 Jan 2013 02:32:28 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [192.12.16.226] Content-Type: multipart/signed; boundary="Apple-Mail=_7709E60E-0ED9-44C9-8D17-BD710D232D83"; protocol="application/pkcs7-signature"; micalg=sha1 MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrMIsWRmVeSWpSXmKPExsXSICQ+U9dhLkOgwdGN3Bade/ayOTB6bPxw nDGAMYrLJiU1J7MstUjfLoEr486TZraCNWUV156eZG1g/JnVxcjJISFgInF2/iN2CFtM4sK9 9WxdjFwcQgI7mSTuL2tmgXB2M0pcvvmCGaSKTUBe4vD2B2C2iIClxK0FH1lAbGEBJYn526Yy QcSVJbadX8UCYRtJrDixnw3EZhFQlehoOApWwytgJbH2/jSwuJBAgMSBpUdZQWxOgUCJ3iOz wXoZgS76fmoNWD2zgLjErSfzmSAuFZF4ePE0G4QtKvHy8T9WCFtRYl3TX3aQo5kFpjBK/F1+ jh1imaDEyZlPWCYwisxCMmsWsrpZSOpmMXIAJZIkvjyLhajXlli28DUzhG0g8bTzFSumuL7E m3dzmCBsU4nXRz8yQtjWEjN+HWSDsBUlpnQ/ZF/AyL2KUSapKDM9oyQ3MTNHFxahJnrFRZl6 yfm5mxjBURvSuoNxxR7DQ4wCHIxKPLxOjgyBQqyJZcWVuYcYJTiYlUR41eqBQrwpiZVVqUX5 8UWlOanFhxilOViUxHm/mfP7CwmkJ5akZqemFqQWwWSZODilGhit5ngorVkf0Lnu8cMMv7eO NXbXFWQMHZbZyU9lyFA/p3j8qFn989mJ0a/czqxNWXZ53aMQvv87bCdINxlMCt8VPdF+RWvu 7muuXXk5/dLZ/znW2PjPNH4uvknkwWo2n1/B+97a+0t9dej+9c/acM3GdLvVmtOv17021lko t7JPp2Naxp/+l4lKLMUZiYZazEXFiQBt7L2D1gIAAA== X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_7709E60E-0ED9-44C9-8D17-BD710D232D83 Content-Type: multipart/alternative; boundary="Apple-Mail=_21061C27-1840-4268-90C9-CC2B83119005" --Apple-Mail=_21061C27-1840-4268-90C9-CC2B83119005 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Wouldn't an update handler work that's designed to do something like an = UPSERT? http://wiki.apache.org/couchdb/Document_Update_Handlers Not exactly a bulk api=85 but I think it would do what you are asking... Jim Klo Senior Software Engineer Center for Software Engineering SRI International t. @nsomnac On Jan 22, 2013, at 3:14 PM, Mark Wakabayashi = wrote: > Hi, >=20 > I need a way to automatically ensure that certain documents exist with > pre-specified content as part of a script. That is, I have a file with = a > set of documents, and I want to run a script that will either insert = them > if they don't exist, or else ensure that the documents in the database > exactly match what I have in the file. Any documents not specified in = the > file should remain untouched. >=20 > The best way I've found so far to do this has been to: > *have the documents in a file in the format described in > http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API > *delete each document. Something like: > for ID in `grep "_id" $FILE | sed 's/.*"_id":"\([^"]*\)".*/\1/'`; do > REV=3D`curl -X GET --silent http://${COUCH_HOST}/foo/${ID} | sed > 's/.*"_rev":"\([^"]*\)".*/\1/'` > curl -X DELETE http://${COUCH_HOST}/foo/${ID}?rev=3D${REV} > done > *insert the documents as they are in the file using _bulk_docs >=20 > Is there a better way to do this? >=20 >=20 > This seems to work most of the time, but I'm having intermittent = failures > where the _bulk_docs reports success but doesn't actually insert the > documents. If I run the script repeatedly, the bulk insert will > occasionally report that the documents have been inserted with = revision 1, > and the documents are then reported as 'deleted'. That is, the bulk = insert > returns something like: > = [{"ok":true,"id":"bar","rev":"1-243f8f87ed4b0abe0ef00c725d346e07"},...] > and document "bar" remains deleted, where normally it would return = much > higher revisions, like > = [{"ok":true,"id":"bar","rev":"229-98ff2bb4ad8754cb254ef2e0392d6ab0"},...] > and "bar" would be available. >=20 > Running the script again fixes the problem. Is this a bug in bulk > insertion? Are there any relevant limitations of bulk insertion that I > should be aware of? >=20 > Thanks in advance, > Mark --Apple-Mail=_21061C27-1840-4268-90C9-CC2B83119005 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 http://wi= ki.apache.org/couchdb/Document_Update_Handlers

Not exactly a bulk api=85 but I think it would do what you are = asking...


Jim Klo
Senior Software Engineer
Center for Software Engineering
SRI = International
t. = @nsomnac

On Jan 22, 2013, at 3:14 PM, Mark Wakabayashi <mark@moshymoshy.com> = wrote:

Hi,

I need a way to automatically ensure that = certain documents exist with
pre-specified content as part of a = script. That is, I have a file with a
set of documents, and I want to = run a script that will either insert them
if they don't exist, or = else ensure that the documents in the database
exactly match what I = have in the file. Any documents not specified in the
file should = remain untouched.

The best way I've found so far to do this has = been to:
*have the documents in a file in the format described = in
http://wiki= .apache.org/couchdb/HTTP_Bulk_Document_API
*delete each document. = Something like:
for ID in `grep "_id" $FILE | sed = 's/.*"_id":"\([^"]*\)".*/\1/'`; do
 REV=3D`curl -X GET --silent = http://${COUCH_HOST}/foo/${ID} | = sed
's/.*"_rev":"\([^"]*\)".*/\1/'`
 curl -X DELETE = http://${COUCH_HOST}/foo/${ID}?rev=3D${REV}
done
*insert the = documents as they are in the file using _bulk_docs

Is there a = better way to do this?


This seems to work most of the time, = but I'm having intermittent failures
where the _bulk_docs reports = success but doesn't actually insert the
documents. If I run the = script repeatedly, the bulk insert will
occasionally report that the = documents have been inserted with revision 1,
and the documents are = then reported as 'deleted'. That is, the bulk insert
returns = something = like:
[{"ok":true,"id":"bar","rev":"1-243f8f87ed4b0abe0ef00c725d346e07"= },...]
and document "bar" remains deleted, where normally it would = return much
higher revisions, = like
[{"ok":true,"id":"bar","rev":"229-98ff2bb4ad8754cb254ef2e0392d6ab0= "},...]
and "bar" would be available.

Running the script again = fixes the problem. Is this a bug in bulk
insertion? Are there any = relevant limitations of bulk insertion that I
should be aware = of?

Thanks in = advance,
Mark

= --Apple-Mail=_21061C27-1840-4268-90C9-CC2B83119005-- --Apple-Mail=_7709E60E-0ED9-44C9-8D17-BD710D232D83 Content-Disposition: attachment; filename="smime.p7s" Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJ/zCCBO0w ggRWoAMCAQICEDJaYyWeLlo372z82sws71YwDQYJKoZIhvcNAQEFBQAwgdgxCzAJBgNVBAYTAlVT MRowGAYDVQQKExFTUkkgSW50ZXJuYXRpb25hbDEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0 d29yazE7MDkGA1UECxMyVGVybXMgb2YgdXNlIGF0IGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9y cGEgKGMpMDIxMDAuBgNVBAsTJ0NsYXNzIDIgT25TaXRlIEluZGl2aWR1YWwgU3Vic2NyaWJlciBD QTEdMBsGA1UEAxMUU1JJIEludGVybmF0aW9uYWwgQ0EwHhcNMTIxMjIyMDAwMDAwWhcNMTMwNjIw MjM1OTU5WjCBwjEaMBgGA1UEChQRU1JJIEludGVybmF0aW9uYWwxKDAmBgNVBAsUH0luZm9ybWF0 aW9uIFRlY2hub2xvZ3kgU2VydmljZXMxRjBEBgNVBAsTPXd3dy52ZXJpc2lnbi5jb20vcmVwb3Np dG9yeS9DUFMgSW5jb3JwLiBieSBSZWYuLExJQUIuTFREKGMpOTkxEjAQBgNVBAMTCUphbWVzIEts bzEeMBwGCSqGSIb3DQEJARYPamltLmtsb0BzcmkuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEArcAbsKPrj0OY8eP+7cwOoUFxQcaL/voNDjR7tLexnZxcS97AyPjJokX5B4ayC7Mh oqvbgdFXf74mZIwKtoMIpf3SDopp68M2sTbhII4weZzn1yYLnVB8p1dH3h2hZDS+AHDY3qAchch0 C5aGQ/ahgJDAhgEiJ/NCbyFAboIhrDsyRS3XxyIlegbrYh9JYRaWdgoxZ21cdJG2xDUPyv/cLVam XKexUqXFLIKH7KKFFwAmVBj6BhZsjjgbMlC1AF3yV2TppULrSS1tDQdHWh5dL+Mw+QXS1O49W0hZ 7V1xFEXvqVj8SdGOaCAphVIU6RPAg7AEiAMynx/sdF5UMZsJuQIDAQABo4IBRjCCAUIwCQYDVR0T BAIwADCBrAYDVR0gBIGkMIGhMIGeBgtghkgBhvhFAQcXAjCBjjAoBggrBgEFBQcCARYcaHR0cHM6 Ly93d3cudmVyaXNpZ24uY29tL0NQUzBiBggrBgEFBQcCAjBWMBUWDlZlcmlTaWduLCBJbmMuMAMC AQEaPVZlcmlTaWduJ3MgQ1BTIGluY29ycC4gYnkgcmVmZXJlbmNlIGxpYWIuIGx0ZC4gKGMpOTcg VmVyaVNpZ24wCwYDVR0PBAQDAgWgMBEGCWCGSAGG+EIBAQQEAwIHgDBmBgNVHR8EXzBdMFugWaBX hlVodHRwOi8vb25zaXRlY3JsLnZlcmlzaWduLmNvbS9TUklJbnRlcm5hdGlvbmFsSW5mb3JtYXRp b25UZWNobm9sb2d5U2VydmljZXMvTGF0ZXN0Q1JMMA0GCSqGSIb3DQEBBQUAA4GBAMShtqQuVGbd YU5ZlqbBxf9ejMIfFgBIn7wubcTIyCAN2nIiDaAqmgS6eE8QiV6A2AchRz87CTQXDH+lkHaSgMiI MtW+n78v+kOLauCqau5iQ2vOwmBx+ypTVu1yV1JQ8GhwQl8hbHaKsVULtlBnPlbCMCgtf4XDBHOL CDLxEJN5MIIFCjCCBHOgAwIBAgIQFnwAoITZjkQu1m3KBG96NzANBgkqhkiG9w0BAQUFADCBwTEL MAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTwwOgYDVQQLEzNDbGFzcyAyIFB1 YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gRzIxOjA4BgNVBAsTMShjKSAx OTk4IFZlcmlTaWduLCBJbmMuIC0gRm9yIGF1dGhvcml6ZWQgdXNlIG9ubHkxHzAdBgNVBAsTFlZl cmlTaWduIFRydXN0IE5ldHdvcmswHhcNMDIwOTIzMDAwMDAwWhcNMTMxMjMxMjM1OTU5WjCB2DEL MAkGA1UEBhMCVVMxGjAYBgNVBAoTEVNSSSBJbnRlcm5hdGlvbmFsMR8wHQYDVQQLExZWZXJpU2ln biBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2UgYXQgaHR0cHM6Ly93d3cudmVy aXNpZ24uY29tL3JwYSAoYykwMjEwMC4GA1UECxMnQ2xhc3MgMiBPblNpdGUgSW5kaXZpZHVhbCBT dWJzY3JpYmVyIENBMR0wGwYDVQQDExRTUkkgSW50ZXJuYXRpb25hbCBDQTCBnzANBgkqhkiG9w0B AQEFAAOBjQAwgYkCgYEAzvnUwmuZmBSSAFVb0qoC0hhUL1a6f+AIHw5UpxW5oRTjsDtUzsCa+6Yg GvKUlisrnI/tPZFrupvHVNQjRj05fhHiABFinwlnCA7J80x3gZlBMwHrgoKYribJ1GTVmc1R0FmA B4KYzBeZjJZiNpqLEsEb0ORdzJYb2/UZazjL/fkCAwEAAaOCAegwggHkMBIGA1UdEwEB/wQIMAYB Af8CAQAwRAYDVR0gBD0wOzA5BgtghkgBhvhFAQcXAjAqMCgGCCsGAQUFBwIBFhxodHRwczovL3d3 dy52ZXJpc2lnbi5jb20vcnBhMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwudmVyaXNpZ24u Y29tL3BjYTItZzIuY3JsMAsGA1UdDwQEAwIBBjARBglghkgBhvhCAQEEBAMCAQYwKAYDVR0RBCEw H6QdMBsxGTAXBgNVBAMTEFByaXZhdGVMYWJlbDItODIwHQYDVR0OBBYEFC1OfgnwbUVBEaxx2j87 9iZKf2RkMIHoBgNVHSMEgeAwgd2hgcekgcQwgcExCzAJBgNVBAYTAlVTMRcwFQYDVQQKEw5WZXJp U2lnbiwgSW5jLjE8MDoGA1UECxMzQ2xhc3MgMiBQdWJsaWMgUHJpbWFyeSBDZXJ0aWZpY2F0aW9u IEF1dGhvcml0eSAtIEcyMTowOAYDVQQLEzEoYykgMTk5OCBWZXJpU2lnbiwgSW5jLiAtIEZvciBh dXRob3JpemVkIHVzZSBvbmx5MR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrghEAuS9g zIifoXpGCbhbcGyKrzANBgkqhkiG9w0BAQUFAAOBgQBocPsx9foGtLlCL8coGlfjYx8GhbDYbdQ3 8w0P/BIw4D49KhAocMcivLESZiV8YYYFFx+ozAPtg0j0knx+tcdeDvWmSefavP+aKlRhpAWk5Z+n c34jLXdw9/+6WveM/OQQbPbd8asD6BsLcFlRm68KZY8kk7SjlsP1S6rQBiCX8jGCBHswggR3AgEB MIHtMIHYMQswCQYDVQQGEwJVUzEaMBgGA1UEChMRU1JJIEludGVybmF0aW9uYWwxHzAdBgNVBAsT FlZlcmlTaWduIFRydXN0IE5ldHdvcmsxOzA5BgNVBAsTMlRlcm1zIG9mIHVzZSBhdCBodHRwczov L3d3dy52ZXJpc2lnbi5jb20vcnBhIChjKTAyMTAwLgYDVQQLEydDbGFzcyAyIE9uU2l0ZSBJbmRp dmlkdWFsIFN1YnNjcmliZXIgQ0ExHTAbBgNVBAMTFFNSSSBJbnRlcm5hdGlvbmFsIENBAhAyWmMl ni5aN+9s/NrMLO9WMAkGBSsOAwIaBQCgggJiMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJ KoZIhvcNAQkFMQ8XDTEzMDEyNDAyMzIzMVowIwYJKoZIhvcNAQkEMRYEFDawvmJ1PBLus51LRpb6 1GQYSaYDMIH+BgkrBgEEAYI3EAQxgfAwge0wgdgxCzAJBgNVBAYTAlVTMRowGAYDVQQKExFTUkkg SW50ZXJuYXRpb25hbDEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazE7MDkGA1UECxMy VGVybXMgb2YgdXNlIGF0IGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9ycGEgKGMpMDIxMDAuBgNV BAsTJ0NsYXNzIDIgT25TaXRlIEluZGl2aWR1YWwgU3Vic2NyaWJlciBDQTEdMBsGA1UEAxMUU1JJ IEludGVybmF0aW9uYWwgQ0ECEDJaYyWeLlo372z82sws71YwggEABgsqhkiG9w0BCRACCzGB8KCB 7TCB2DELMAkGA1UEBhMCVVMxGjAYBgNVBAoTEVNSSSBJbnRlcm5hdGlvbmFsMR8wHQYDVQQLExZW ZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2UgYXQgaHR0cHM6Ly93 d3cudmVyaXNpZ24uY29tL3JwYSAoYykwMjEwMC4GA1UECxMnQ2xhc3MgMiBPblNpdGUgSW5kaXZp ZHVhbCBTdWJzY3JpYmVyIENBMR0wGwYDVQQDExRTUkkgSW50ZXJuYXRpb25hbCBDQQIQMlpjJZ4u WjfvbPzazCzvVjANBgkqhkiG9w0BAQEFAASCAQCVoCixGZ4+Q72zrk+fLzFoS/6bH2Nms+TesQlb 6B8ac2daSW18vjTPuOYTDabzWHBRAxIQGTR5OFC0WLhAZk4lnGdo7c7gktT90PCuluZ3idafnhH0 6M3Lsq0TbLZTr1Q43oQJqiqjWajeR+RNr77qHPYehYlCaGLmSandH3qjGVCb09xRsODYCy1MmQ9W yau95f0EF6emE5cDub4AZ9ODxygSz7M0ZiazTKj13vODENs7HwQmN8LwLMkVpBn/G1tq7hGhjpH8 fJbIBaZSZvxVep/1uWaJ9lh3ya9ERMGMJ5ktJDQ+nUKDp5GL4bY/WlakghZseYkV1Lr1cX3DfSVX AAAAAAAA --Apple-Mail=_7709E60E-0ED9-44C9-8D17-BD710D232D83--