couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Gonzalez <gonva...@gonvaled.com>
Subject Re: bulk update failing when document has attachments?
Date Wed, 11 Dec 2013 13:07:02 GMT
Thanks for your support. Problem solved.

The explanation: now I was removing correctly the data-uri prefix, but the
base64 encoded test data that I was using was actually not valid base64
data (it was manually defined, just for testing). It turns out that couchdb
decodes (of course! hindsight is everything) the data it receives in the
attachment, and because it was not real base64 encoded data, it complains.

The badarg reply is very cryptic, nevertheless. I still do not know what
that means.

Thanks and regards,
Daniel


On Wed, Dec 11, 2013 at 1:58 PM, Daniel Gonzalez <gonvaled@gonvaled.com>wrote:

> It is not an artifact: I am taking that from the couchdb documentation.
>
> And according to Alexander Shorin forward slashes **really** need to be
> escaped in json. But it is not me who must do that, but the library
> converting the python objects to couchdb, so that can not be my problem.
>
> Now I am left with a badarg exception, which I can not relate to my input
> data:
>
> Exception > Problems updating list of documents (length = 1): (500,
> ('badarg', '46'))
>
> What does that '46' mean?
>
>
> On Wed, Dec 11, 2013 at 1:47 PM, Robert Newson <rnewson@apache.org> wrote:
>
>> I think your "image\/png" is just an artifact of your printing method,
>> you don't need to escape the forward slash in content_type, see
>> example below;
>>
>>
>> {"_id":"doc1","_rev":"1-96e2a6c78b8bfb227e79e1fbb16873f9","_attachments":{"att1":{"content_type":"image/png","revpos":1,"digest":"md5-XUFAKrxLKna5cZ2REBfFkg==","length":5,"stub":true}}}
>>
>> B.
>>
>>
>> On 11 December 2013 12:35, Daniel Gonzalez <gonvaled@gonvaled.com> wrote:
>> > That would work *only* for that prefix (data:image/png;base64,), or any
>> > prefix which happens to have the same length. Not very robust.
>> >
>> > I just discovered that the data coming from the front-end comes in
>> data-uri
>> > format (rfc2397). This should handle any rfc2397 prefix:
>> > http://stackoverflow.com/a/20518589/647991 (maybe buggy, just
>> implemented).
>> >
>> > Another question: even after removing the data-uri prefix, I am still
>> > getting problems. I think my content type is not right.
>> >
>> > Must content_type be escaped? That is:
>> >
>> > 'content_type': 'image/png', -> 'content_type': 'image\/png',
>> >
>> > The only reference I see to that is an example here:
>> > http://wiki.apache.org/couchdb/HTTP_Document_API#Inline_Attachments
>> >
>> > But no real explanation of why. It seems no other strings must be
>> escaped
>> > for couchdb. The only requirement that couchdb seems to impose on top of
>> > json is that the data in the attachment must be in base64 format.
>> >
>> > But now it seems that the content_type must escape the slashes (/).
>> Why? It
>> > does not seem to be a json feature: slashes are fine in any json
>> string. So
>> > what is that?
>> >
>> > I would like to know the specificacion for the format expected for
>> > content_type. Does that have a name? I am calling it "escaped
>> mediatype".
>> > Is it part of a more generic escaping process expected by couchdb, or
>> only
>> > the content_type is affected? Is there an official name for that?
>> >
>> >
>> > On Wed, Dec 11, 2013 at 12:18 PM, Johannes Jörg Schmidt <
>> > schmidt@netzmerk.com> wrote:
>> >
>> >> data.slice(22)
>> >>
>> >> 2013/12/11 Daniel Gonzalez <gonvaled@gonvaled.com>:
>> >> > Thanks, I just realized about this. The base64 is coming from the
>> >> > javascript frontend (chose file in a form). So I need to remove the
>> >> > prefix "data:image/png;base64,".
>> >> > Not sure how to do this without rolling my own regexes though.
>> >> >
>> >> >
>> >> > On Wed, Dec 11, 2013 at 12:01 PM, Alexander Shorin <kxepal@gmail.com
>> >
>> >> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> _attachments data should be valid base64 encoded string, while
you
>> have:
>> >> >> >
>> >> >>
>> >>
>> data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAM8AAADkCAIAAACwiOf9AAAAA3NCSVQICAjb4U/gAAAgAElEQVR4nO...
>> >> >>
>> >> >> Chars : and , are invalid for base64.
>> >> >> --
>> >> >> ,,,^..^,,,
>> >> >>
>> >> >>
>> >> >> On Wed, Dec 11, 2013 at 2:49 PM, Daniel Gonzalez <
>> gonvaled@gonvaled.com
>> >> >
>> >> >> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > (SO reference: http://stackoverflow.com/q/20516980/647991.
I post
>> >> there
>> >> >> > because formatting makes things much easier to read, replies
/
>> >> comments
>> >> >> are
>> >> >> > well organized, and the up/downvote mechanism works)
>> >> >> >
>> >> >> > I am performing the following operation:
>> >> >> >
>> >> >> > 1. Prepare some documents: `docs = [ doc1, doc2, ... ]`. The
>> documents
>> >> >> have
>> >> >> > *maybe* attachments
>> >> >> > 2. I `POST` to `_bulk_docs` the list of documents
>> >> >> > 3. I get an `Exception > Problems updating list of documents
>> (length =
>> >> >> 1):
>> >> >> > (500, ('badarg', '58'))`
>> >> >> >
>> >> >> > My `bulk_docs` is (in this case just one):
>> >> >> >
>> >> >> >     [   {   '_attachments': {   'image.png': {   'content_type':
>> >> >> > 'image/png',
>> >> >> >                                                  'data':
>> >> >> >
>> >> >>
>> >>
>> 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAM8AAADkCAIAAACwiOf9AAAAA3NCSVQICAjb4U/gAAAgAElEQVR4nO...'}},
>> >> >> >             '_id': '08b8fc66-cd90-47a1-9053-4f6fefabdfe3',
>> >> >> >             '_rev': '15-ff3d0e8baa56e5ad2fac4937264fb3f6',
>> >> >> >             'docmeta': {   'created': '2013-10-01
>> 14:48:24.311257',
>> >> >> >                            'updated': [   '2013-10-01
>> >> 14:48:24.394157',
>> >> >> >                                           '2013-12-11
>> >> 08:19:47.271812',
>> >> >> >                                           '2013-12-11
>> >> 08:25:05.662546',
>> >> >> >                                           '2013-12-11
>> >> 10:38:56.116145']},
>> >> >> >             'org_id': 45345,
>> >> >> >             'outputs_id': None,
>> >> >> >             'properties': {   'auto-t2s': False,
>> >> >> >                               'content_type': 'image/png',
>> >> >> >                               'lang': 'es',
>> >> >> >                               'name': 'dfasdfasdf',
>> >> >> >                               'text': 'erwerwerwrwerwr'},
>> >> >> >             'subtype': 'voicemail-st',
>> >> >> >             'tags': ['RRR-ccc-dtjkqx'],
>> >> >> >             'type': 'recording'}]
>> >> >> >
>> >> >> > This is the detailed exception:
>> >> >> >
>> >> >> >     Traceback (most recent call last):
>> >> >> >       File "portal_support_ut.py", line 470, in test_UpdateDoc
>> >> >> >         self.ps.UpdateDoc(self.org_id, what, doc_id, new_data)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/ps/complex_ops.py",
>> >> >> > line 349, in UpdateDoc
>> >> >> >         success, doc = database.UpdateDoc(doc_id, new_data)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/updater.py",
>> >> >> > line 38, in UpdateDoc
>> >> >> >         res = self.SaveDoc(doc_id, doc)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/saver.py",
>> >> >> > line 88, in SaveDoc
>> >> >> >         else      : self.bulk_append(doc, flush, update_revision)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/bulker.py",
>> >> >> > line 257, in bulk_append
>> >> >> >         if force_send or flush or not self.timer.use_timer
:
>> >> >> > self.BulkSend(show_progress=True)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/bulker.py",
>> >> >> > line 144, in BulkSend
>> >> >> >         results = self.UpdateDocuments(self.bulk)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/projects/new-wavilon-portal/python_modules/wav/cdb/core/bulker.py",
>> >> >> > line 67, in UpdateDocuments
>> >> >> >         results = self.db.update(bulkdocs)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/client.py",
>> >> >> > line 764, in update
>> >> >> >         _, _, data = self.resource.post_json('_bulk_docs',
>> >> body=content)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py",
>> >> >> > line 527, in post_json
>> >> >> >         **params)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py",
>> >> >> > line 546, in _request_json
>> >> >> >         headers=headers, **params)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py",
>> >> >> > line 542, in _request
>> >> >> >         credentials=self.credentials)
>> >> >> >       File
>> >> >> >
>> >> >>
>> >>
>> "/home/gonvaled/.virtualenvs/python2.7.3-wavilon1/local/lib/python2.7/site-packages/couchdb/http.py",
>> >> >> > line 398, in request
>> >> >> >         raise ServerError((status, error))
>> >> >> >     ServerError: (500, ('badarg', '58'))
>> >> >> >
>> >> >> >
>> >> >> > What does that `badarg` mean? Is it possible to send attachments
>> when
>> >> >> doing
>> >> >> > `_bulk_docs`?
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Daniel
>> >> >>
>> >>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message