Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of Viktor.Szabo@morganstanley.com
 designates 205.228.53.69 as permitted sender)
From: "Szabo, Viktor" <Viktor.Szabo@morganstanley.com>
To: <user@couchdb.apache.org>
Content-Class: urn:content-classes:message
Importance: normal
Priority: normal
Subject: RE: possible compact bug in 1.1.1
Thread-Topic: possible compact bug in 1.1.1
thread-index: 
 AcypA5vr6+QWsbbdRX6tO3a1wVdWHQATN7mAACKWF4AAAJKpAAAA33mQAAEfMYARi2BDwA==
Date: Mon, 20 Feb 2012 21:16:12 +0000
Message-ID: <9032450E6B3D8244858652FFFD6E29BC10491A@OZWEX0202N2.msad.ms.com>
References: 
 <9032450E6B3D8244858652FFFD6E29BC0294FC@OZWEX0202N2.msad.ms.com><CAJ_m3YC-jVPd+HC2xi6sz5s0JdecZCV7jY+p355wXJKLqUNzvg@mail.gmail.com><9032450E6B3D8244858652FFFD6E29BC02A825@OZWEX0202N2.msad.ms.com><CAMWZAjnuWrMtRhKH83eFNORWNiP4p_ReTZP+U5dkprJ_h5e-+w@mail.gmail.com><9032450E6B3D8244858652FFFD6E29BC02A8A0@OZWEX0202N2.msad.ms.com>
 <CAMWZAjn3pHFivokF+zs7uebSHtLqY9BGLAtwbP5_BqoUHtcXBA@mail.gmail.com>
In-Reply-To: 
 <CAMWZAjn3pHFivokF+zs7uebSHtLqY9BGLAtwbP5_BqoUHtcXBA@mail.gmail.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi,

An application we have is using bulk writes to insert objects into the =
database, and occasionally some
documents get lost. It turns out this happens for the very same reason =
described below. If we're
re-insering a doc using an id that has been deleted earlier, if the =
contents match the payload that was
saved when the id got used for the first time, the doc remains in =
"deleted" state and the client
gets a misleading response. This only happens when the database is =
compacted after the delete operation
is executed.

I still feel that this is pretty nasty - what do you guys think?

Looking at the reply coming from _bulk_docs, there's no way I can tell =
if I can read back my doc from the database or not.

Below are the steps to reproduce.

Thanks,
Viktor

$ curl -X POST -H "Content-type: application/json" =
http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key" =
: "value" } ] }'
[{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
=20
$ curl -X GET http://server/database/bug
{"_id":"bug","_rev":"1-59414e77c768bc202142ac82c2f129de","key":"value"}
=20
$ curl -X DELETE =
http://server/database/bug?rev=3D1-59414e77c768bc202142ac82c2f129de
{"ok":true,"id":"bug","rev":"2-9b2e3bcc3752a3a952a3570b2ed4d27e"}
=20
$ curl -X GET http://server/database/bug
{"error":"not_found","reason":"deleted"}
=20
$ curl -X POST -H "Content-type: application/json" =
http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key" =
: "value" } ] }'
[{"id":"bug","rev":"3-82b9390af5b7c003e03c0dd7e6aac45a"}]
=20
$ curl -X GET http://server/database/bug
{"_id":"bug","_rev":"3-82b9390af5b7c003e03c0dd7e6aac45a","key":"value"}
=20
$ curl -X DELETE =
http://server/database/bug?rev=3D3-82b9390af5b7c003e03c0dd7e6aac45a
{"ok":true,"id":"bug","rev":"4-1f98d35af7fd5ce66197980f295e5dba"}
=20
$  curl -X POST -H "Content-type: application/json" =
http://server/database/_compact
{"ok":true}
=20
$ curl -X GET http://server/database/bug
{"error":"not_found","reason":"deleted"}
=20
$ curl -X POST -H "Content-type: application/json" =
http://server/database/_bulk_docs -d '{"docs" : [ { "_id" : "bug", "key" =
: "value" } ] }'
[{"id":"bug","rev":"1-59414e77c768bc202142ac82c2f129de"}]
=20
$ curl -X GET http://server/database/bug
{"error":"not_found","reason":"deleted"}
=20

-----Original Message-----
From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]=20
Sent: 23 November 2011 14:38
To: user@couchdb.apache.org
Subject: Re: possible compact bug in 1.1.1

When a new document is created, the response is 201, not 200.
If the posted document is identical to the one saved, the revision is =
the same (otherwise it would have been a conflict). So the revision you =
have is not obsolete.

Marcello

2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
<Viktor.Szabo@morganstanley.com>:
> I'd rather know about the fact that I haven't just successfully=20
> created a doc, but re-submitted a revision that was already known -=20
> and is already obsolete as revisions with higher version numbers =
already exist.
>
> Cheers,
> Viktor
>
> -----Original Message-----
> From: Marcello Nuccio [mailto:marcello.nuccio@gmail.com]
> Sent: 23 November 2011 13:41
> To: user@couchdb.apache.org
> Subject: Re: possible compact bug in 1.1.1
>
> Can you elaborate on why you want a conflict?
> I find it confusing to have a conflict when, in fact, there can't be =
any conflict since nothing has changed.
>
> Marcello
>
> 2011/11/23 Szabo, Viktor (Enterprise Infrastructure)
> <Viktor.Szabo@morganstanley.com>:
>> Thanks Paul, this makes sense.
>>
>> If it counts, I vote for forcing a conflict ;)
>>
>> Cheers,
>> Viktor
>>
>> -----Original Message-----
>> From: Paul Davis [mailto:paul.joseph.davis@gmail.com]
>> Sent: 22 November 2011 20:54
>> To: user@couchdb.apache.org
>> Subject: Re: possible compact bug in 1.1.1
>>
>>
>> Your example here is actually hitting a very specific edge case as =
demonstrated by Marcello's test. As of many versions ago, revisions are =
generated using a hashing scheme of the document contents. In your =
particular case the requests you're issuing contain the same identical =
data in such a way that CouchDB will generate a revision of the doc.
>>
>> Given this, we then have to look at how this plays into replication.
>> Basically, when we merge the revision trees we get to the case where =
it's "oh, we already have this version, cool" because we do already have =
this version.
>>
>> Whether or not that behavior is best, or if we should force a =
conflict if we don't add a leaf during a write is another question. In =
other words, the system is working fine, but this particular behavior =
can be a bit unexpected.
>>
>> ---------------------------------------------------------------------
>> -
>> ----
>> NOTICE: Morgan Stanley is not acting as a municipal advisor and the =
opinions or views contained herein are not intended to be, and do not =
constitute, advice within the meaning of Section 975 of the Dodd-Frank =
Wall Street Reform and Consumer Protection Act. If you have received =
this communication in error, please destroy all electronic and paper =
copies and notify the sender immediately. Mistransmission is not =
intended to waive confidentiality or privilege. Morgan Stanley reserves =
the right, to the extent permitted under applicable law, to monitor =
electronic communications. This message is subject to terms available at =
the following link: http://www.morganstanley.com/disclaimers. If you =
cannot access these links, please notify us by reply message and we will =
send the contents to you. By messaging with Morgan Stanley you consent =
to the foregoing.
>
> ----------------------------------------------------------------------
> ----
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the =
opinions or views contained herein are not intended to be, and do not =
constitute, advice within the meaning of Section 975 of the Dodd-Frank =
Wall Street Reform and Consumer Protection Act. If you have received =
this communication in error, please destroy all electronic and paper =
copies and notify the sender immediately. Mistransmission is not =
intended to waive confidentiality or privilege. Morgan Stanley reserves =
the right, to the extent permitted under applicable law, to monitor =
electronic communications. This message is subject to terms available at =
the following link: http://www.morganstanley.com/disclaimers. If you =
cannot access these links, please notify us by reply message and we will =
send the contents to you. By messaging with Morgan Stanley you consent =
to the foregoing.

-------------------------------------------------------------------------=
-
NOTICE: Morgan Stanley is not acting as a municipal advisor and the =
opinions or views contained herein are not intended to be, and do not =
constitute, advice within the meaning of Section 975 of the Dodd-Frank =
Wall Street Reform and Consumer Protection Act. If you have received =
this communication in error, please destroy all electronic and paper =
copies and notify the sender immediately. Mistransmission is not =
intended to waive confidentiality or privilege. Morgan Stanley reserves =
the right, to the extent permitted under applicable law, to monitor =
electronic communications. This message is subject to terms available at =
the following link: http://www.morganstanley.com/disclaimers. If you =
cannot access these links, please notify us by reply message and we will =
send the contents to you. By messaging with Morgan Stanley you consent =
to the foregoing.