Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
MIME-Version: 1.0
In-Reply-To: 
 <CABvT1DG9x-6rbntQmiiaFPXBVHSV8meMvjAkaA06VwFRtL09Kw@mail.gmail.com>
References: 
 <CAAOi-OFBAN3UF52wZcr_wffFNv_RTS6QN-cGfiNCwUpqXaeYnw@mail.gmail.com>
 <CAHdjipJWuh2sOUwNqJ-=An-GxBUUyTQqEBQFdhYuyVBy=iP=1g@mail.gmail.com>
 <CAAOi-OE+PthzC6DjR1+cL=OKw3TA0Kg_dEsY0YPR02Jb8u+FYg@mail.gmail.com>
 <CAO5PmOpTwLa5Wk2b2JWw0ngqZ8NiUTQYP=CWBsOo4KGGceMx0A@mail.gmail.com>
 <CAAOi-OEZhanhJMZe5QWJRJb+uMQYnsTotD-Q4SiYtv3KNMP_Bw@mail.gmail.com>
 <CABvT1DHTYbjhFbPZ8NWDmk08AJBOQ+X_Vor5CaT1J1H_dAzkzw@mail.gmail.com>
 <CAAOi-OF7KWgeoez1rA+x3izuxE+vtyH4zRYu2dF+FMBawf_QVg@mail.gmail.com>
 <CABvT1DHKngTU0XwepPWwDKaF6HOugr0b5s28Kwy_bSeXATJjZQ@mail.gmail.com>
 <CABvT1DG9x-6rbntQmiiaFPXBVHSV8meMvjAkaA06VwFRtL09Kw@mail.gmail.com>
From: Daniel Gonzalez <gonvaled@gonvaled.com>
Date: Wed, 11 Dec 2013 14:18:22 +0100
Message-ID: 
 <CAAOi-OEEjtX=1pYMkdQ-BMLz5UkB3nEkU-riCWi4X7r5Sg13pw@mail.gmail.com>
Subject: Re: bulk update failing when document has attachments?
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=089e01494a16fb84ce04ed4211ca

--089e01494a16fb84ce04ed4211ca
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Funny that you do not need to escape it. The spec says you should:

char
any-Unicode-character-
    except-"-or-\-or-
    control-character
\"
\\
\/
\b
\f
\n
\r
\t
\u four-hex-digits

Anyway, my problem has been solved. I am not escaping anything in the
content_type: the json library is problably doing that. What I need to do
is to attach real base64 encoded data, which has solved my problem.


On Wed, Dec 11, 2013 at 2:15 PM, Robert Newson <rnewson@apache.org> wrote:

> =E2=9E=9C  ~  curl localhost:5984/db1/_bulk_docs
> -Hcontent-type:application/json -d
> '{"docs":[{"_attachments":{"foo":{"data":"aGVsbG8=3D"}}}]}'
>
> [{"ok":true,"id":"b5d2060479624e483a8fe4747f001dbe","rev":"1-12c665c499a5=
25a3a1a9ad35c90604a1"}]
>
> =E2=9E=9C  ~  curl localhost:5984/db1/b5d2060479624e483a8fe4747f001dbe
>
> {"_id":"b5d2060479624e483a8fe4747f001dbe","_rev":"1-12c665c499a525a3a1a9a=
d35c90604a1","_attachments":{"foo":{"content_type":"application/octet-strea=
m","revpos":1,"digest":"md5-XUFAKrxLKna5cZ2REBfFkg=3D=3D","length":5,"stub"=
:true}}}
>
> =E2=9E=9C  ~  curl localhost:5984/db1/b5d2060479624e483a8fe4747f001dbe/fo=
o
> hello%
>
> Maybe you left escaped newlines in your base64 input?
>
> B.
>
>
> On 11 December 2013 13:11, Robert Newson <rnewson@apache.org> wrote:
> > http://json.org/string.gif talks escaping back slash, not forward
> > slash. The PDF page 194 talks about escaping forward slash within a
> > RegExp statement in Javascript, which is not JSON.
> >
> > B.
> >
> >
> > On 11 December 2013 12:58, Daniel Gonzalez <gonvaled@gonvaled.com>
> wrote:
> >> It is not an artifact: I am taking that from the couchdb documentation=
.
> >>
> >> And according to Alexander Shorin forward slashes **really** need to b=
e
> >> escaped in json. But it is not me who must do that, but the library
> >> converting the python objects to couchdb, so that can not be my proble=
m.
> >>
> >> Now I am left with a badarg exception, which I can not relate to my
> input
> >> data:
> >>
> >> Exception > Problems updating list of documents (length =3D 1): (500,
> >> ('badarg', '46'))
> >>
> >> What does that '46' mean?
> >>
> >>
> >> On Wed, Dec 11, 2013 at 1:47 PM, Robert Newson <rnewson@apache.org>
> wrote:
> >>
> >>> I think your "image\/png" is just an artifact of your printing method=
,
> >>> you don't need to escape the forward slash in content_type, see
> >>> example below;
> >>>
> >>>
> >>>
> {"_id":"doc1","_rev":"1-96e2a6c78b8bfb227e79e1fbb16873f9","_attachments":=
{"att1":{"content_type":"image/png","revpos":1,"digest":"md5-XUFAKrxLKna5cZ=
2REBfFkg=3D=3D","length":5,"stub":true}}}
> >>>
> >>> B.
> >>>
> >>>
> >>> On 11 December 2013 12:35, Daniel Gonzalez <gonvaled@gonvaled.com>
> wrote:
> >>> > That would work *only* for that prefix (data:image/png;base64,), or
> any
> >>> > prefix which happens to have the same length. Not very robust.
> >>> >
> >>> > I just discovered that the data coming from the front-end comes in
> >>> data-uri
> >>> > format (rfc2397). This should handle any rfc2397 prefix:
> >>> > http://stackoverflow.com/a/20518589/647991 (maybe buggy, just
> >>> implemented).
> >>> >
> >>> > Another question: even after removing the data-uri prefix, I am sti=
ll
> >>> > getting problems. I think my content type is not right.
> >>> >
> >>> > Must content_type be escaped? That is:
> >>> >
> >>> > 'content_type': 'image/png', -> 'content_type': 'image\/png',
> >>> >
> >>> > The only reference I see to that is an example here:
> >>> > http://wiki.apache.org/couchdb/HTTP_Document_API#Inline_Attachments
> >>> >
> >>> > But no real explanation of why. It seems no other strings must be
> escaped
> >>> > for couchdb. The only requirement that couchdb seems to impose on
> top of
> >>> > json is that the data in the attachment must be in base64 format.
> >>> >
> >>> > But now it seems that the content_type must escape the slashes (/).
> Why?
> >>> It
> >>> > does not seem to be a json feature: slashes are fine in any json
> string.
> >>> So
> >>> > what is that?
> >>> >
> >>> > I would like to know the specificacion for the format expected for
> >>> > content_type. Does that have a name? I am calling it "escaped
> mediatype".
> >>> > Is it part of a more generic escaping process expected by couchdb, =
or
> >>> only
> >>> > the content_type is affected? Is there an official name for that?
> >>> >
> >>> >
> >>> > On Wed, Dec 11, 2013 at 12:18 PM, Johannes J=C3=B6rg Schmidt <
> >>> > schmidt@netzmerk.com> wrote:
> >>> >
> >>> >> data.slice(22)
> >>> >>
> >>> >> 2013/12/11 Daniel Gonzalez <gonvaled@gonvaled.com>:
> >>> >> > Thanks, I just realized about this. The base64 is coming from th=
e
> >>> >> > javascript frontend (chose file in a form). So I need to remove
> the
> >>> >> > prefix "data:image/png;base64,".
> >>> >> > Not sure how to do this without rolling my own regexes though.
> >>> >> >
> >>> >> >
> >>> >> > On Wed, Dec 11, 2013 at 12:01 PM, Alexander Shorin <
> kxepal@gmail.com>
> >>> >> wrote:
> >>> >> >
> >>> >> >> Hi,
> >>> >> >>
> >>> >> >> _attachments data should be valid base64 encoded string, while
> you
> >>> have:
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAM8AAADkCAIAAACwiOf9AAAAA3N=
CSVQICAjb4U/gAAAgAElEQVR4nO...
> >>> >> >>
> >>> >> >> Chars : and , are invalid for base64.
> >>> >> >> --
> >>> >> >> ,,,^..^,,,
> >>> >> >>
> >>> >> >>
> >>> >> >> On Wed, Dec 11, 2013 at 2:49 PM, Daniel Gonzalez <
> >>> gonvaled@gonvaled.com
> >>> >> >
> >>> >> >> wrote:
> >>> >> >> > Hi,
> >>> >> >> >
> >>> >> >> > (SO reference: http://stackoverflow.com/q/20516980/647991. I
> post
> >>> >> there
> >>> >> >> > because formatting makes things much easier to read, replies =
/
> >>> >> comments
> >>> >> >> are
> >>> >> >> > well organized, and the up/downvote mechanism works)
> >>> >> >> >
> >>> >> >> > I am performing the following operation:
> >>> >> >> >
> >>> >> >> > 1. Prepare some documents: `docs =3D [ doc1, doc2, ... ]`. Th=
e
> >>> documents
> >>> >> >> have
> >>> >> >> > *maybe* attachments
> >>> >> >> > 2. I `POST` to `_bulk_docs` the list of documents
> >>> >> >> > 3. I get an `Exception > Problems updating list of documents
> >>> (length =3D
> >>> >> >> 1):
> >>> >> >> > (500, ('badarg', '58'))`
> >>> >> >> >
> >>> >> >> > My `bulk_docs` is (in this case just one):
> >>> >> >> >
> >>> >> >> >     [   {   '_attachments': {   'image.png': {
> 'content_type':
> >>> >> >> > 'image/png',
> >>> >> >> >                                                  'data':
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAM8AAADkCAIAAACwiOf9AAAAA3=
NCSVQICAjb4U/gAAAgAElEQVR4nO...'}},
> >>> >> >> >             '_id': '08b8fc66-cd90-47a1-9053-4f6fefabdfe3',
> >>> >> >> >             '_rev': '15-ff3d0e8baa56e5ad2fac4937264fb3f6',
> >>> >> >> >             'docmeta': {   'created': '2013-10-01
> 14:48:24.311257',
> >>> >> >> >                            'updated': [   '2013-10-01
> >>> >> 14:48:24.394157',
> >>> >> >> >                                           '2013-12-11
> >>> >> 08:19:47.271812',
> >>> >> >> >                                           '2013-12-11
> >>> >> 08:25:05.662546',
> >>> >> >> >                                           '2013-12-11
> >>> >> 10:38:56.116145']},
> >>> >> >> >             'org_id': 45345,
> >>> >> >> >             'outputs_id': None,
> >>> >> >> >             'properties': {   'auto-t2s': False,
> >>> >> >> >                               'content_type': 'image/png',
> >>> >> >> >                               'lang': 'es',
> >>> >> >> >                               'name': 'dfasdfasdf',
> >>> >> >> >                               'text': 'erwerwerwrwerwr'},
> >>> >> >> >             'subtype': 'voicemail-st',
> >>> >> >> >             'tags': ['RRR-ccc-dtjkqx'],
> >>> >> >> >             'type': 'recording'}]
> >>> >> >> >
> >>> >> >> > This is the detailed exception:
> >>> >> >> >
> >>> >> >> >     Traceback (most recent call last):
> >>> >> >> >       File "portal_support_ut.py", line 470, in test_UpdateDo=
c
> >>> >> >> >         self.ps.UpdateDoc(self.org_id, what, doc_id, new_data=
)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/ps/complex=
_ops.py",
> >>> >> >> > line 349, in UpdateDoc
> >>> >> >> >         success, doc =3D database.UpdateDoc(doc_id, new_data)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/u=
pdater.py",
> >>> >> >> > line 38, in UpdateDoc
> >>> >> >> >         res =3D self.SaveDoc(doc_id, doc)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/s=
aver.py",
> >>> >> >> > line 88, in SaveDoc
> >>> >> >> >         else      : self.bulk_append(doc, flush,
> update_revision)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/b=
ulker.py",
> >>> >> >> > line 257, in bulk_append
> >>> >> >> >         if force_send or flush or not self.timer.use_timer :
> >>> >> >> > self.BulkSend(show_progress=3DTrue)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/b=
ulker.py",
> >>> >> >> > line 144, in BulkSend
> >>> >> >> >         results =3D self.UpdateDocuments(self.bulk)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/b=
ulker.py",
> >>> >> >> > line 67, in UpdateDocuments
> >>> >> >> >         results =3D self.db.update(bulkdocs)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/sit=
e-packages/couchdb/client.py",
> >>> >> >> > line 764, in update
> >>> >> >> >         _, _, data =3D self.resource.post_json('_bulk_docs',
> >>> >> body=3Dcontent)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/sit=
e-packages/couchdb/http.py",
> >>> >> >> > line 527, in post_json
> >>> >> >> >         **params)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/sit=
e-packages/couchdb/http.py",
> >>> >> >> > line 546, in _request_json
> >>> >> >> >         headers=3Dheaders, **params)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/sit=
e-packages/couchdb/http.py",
> >>> >> >> > line 542, in _request
> >>> >> >> >         credentials=3Dself.credentials)
> >>> >> >> >       File
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/sit=
e-packages/couchdb/http.py",
> >>> >> >> > line 398, in request
> >>> >> >> >         raise ServerError((status, error))
> >>> >> >> >     ServerError: (500, ('badarg', '58'))
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > What does that `badarg` mean? Is it possible to send
> attachments
> >>> when
> >>> >> >> doing
> >>> >> >> > `_bulk_docs`?
> >>> >> >> >
> >>> >> >> > Thanks,
> >>> >> >> > Daniel
> >>> >> >>
> >>> >>
> >>>
>

--089e01494a16fb84ce04ed4211ca--