incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Waka <waka.m...@gmail.com>
Subject Bulk insert-or-update
Date Tue, 22 Jan 2013 23:19:46 GMT
Hi,

I need a way to automatically ensure that certain documents exist with
pre-specified content as part of a script. That is, I have a file with a
set of documents, and I want to run a script that will either insert them
if they don't exist, or else ensure that the documents in the database
exactly match what I have in the file. Any documents not specified in the
file should remain untouched.

The best way I've found so far to do this has been to:
*have the documents in a file in the format described in
http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API
*delete each document. Something like:
for ID in `grep "_id" $FILE | sed 's/.*"_id":"\([^"]*\)".*/\1/'`; do
  REV=`curl -X GET --silent http://${COUCH_HOST}/foo/${ID} | sed
's/.*"_rev":"\([^"]*\)".*/\1/'`
  curl -X DELETE http://${COUCH_HOST}/foo/${ID}?rev=${REV}
done
*insert the documents as they are in the file using _bulk_docs

Is there a better way to do this?


This seems to work most of the time, but I'm having intermittent failures
where the _bulk_docs reports success but doesn't actually insert the
documents. If I run the script repeatedly, the bulk insert will
occasionally report that the documents have been inserted with revision 1,
and the documents are then reported as 'deleted'. That is, the bulk insert
returns something like:
[{"ok":true,"id":"bar","rev":"1-243f8f87ed4b0abe0ef00c725d346e07"},...]
and document "bar" remains deleted, where normally it would return much
higher revisions, like
[{"ok":true,"id":"bar","rev":"229-98ff2bb4ad8754cb254ef2e0392d6ab0"},...]
and "bar" would be available.

Running the script again fixes the problem. Is this a bug in bulk
insertion? Are there any relevant limitations of bulk insertion that I
should be aware of?

Thanks in advance,
Mark

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message