couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lennart Melzer <l.mel...@tu-bs.de>
Subject Re: POST/PUTing binary Attachments to external Handler
Date Wed, 11 Nov 2009 14:41:53 GMT
Mmmh, so the real problem just seems to be, that the external handling  
of POST/PUT calls does not handle binary data seperately from String  
data. So one has to base64encode those attachments. I'm not really  
good enough at erlang coding (yet) to move the code from the usual  
couchdb handling (for multipart/form-data) to external handlers, but  
that would be really good.

For now I stick with base64 encoding of binary data, which works ok,  
since the produced json object sent to my external ruby script does  
not contain any offending "chars".

Greetings,

Lennart
On Nov 11, 2009, at 8:29 AM, Lennart Melzer wrote:

> Hi Paul,
>
> I double checked the response of my external ruby script.
>
> I reduced the code to the bare minimum (nearly representing what the  
> python example from the wiki looks like)
>
> #!/usr/bin/env ruby
> require 'rubygems'
> require 'json'
>
> def request
>  begin
>    while line = STDIN.readline
>      yield JSON.parse(line)
>    end
>  rescue EOFError
>    return
>  end
> end
>
> def respond code = 200, data = {}, headers = {}
>  output = "#{{:code => code,:body => data, :headers =>  
> headers}.to_json}\n"
>  STDOUT.write output
>  STDOUT.flush
> end
> request do |line|
>  respond 200, "random"
> end
>
> When I do a GET request to the external handler I get "random" back,  
> when I do a POST with a simple form I get "random" back, but when  
> POSTing an image (by curl) as multipart/form-data it breaks with the  
> aforementioned error returning nothing. So since the return value  
> seems to work with other types of request, it can't be my external  
> handler.
>
> Sorry for bothering, but I really want to get this working. Been  
> reading through the code for some time now, but I don't understand  
> why couchdb is even calling send_json in the first place.
>
> Greetings,
>
> Lennart
> On Nov 11, 2009, at 12:11 AM, Paul Davis wrote:
>
>> On Tue, Nov 10, 2009 at 5:56 PM, Lennart Melzer <l.melzer@tu-bs.de>  
>> wrote:
>>> Hi Paul,
>>>
>>> I'm not trying to return arbitrary binary data,I really just try  
>>> to post an
>>> attached file to an external handler. Right now the handler  
>>> returns a static
>>> response which works when posting form data (without binary data  
>>> in it) and
>>> with get request. It doesn't work when posting a binary file. I  
>>> don't think
>>> it's a problem with the handler itself.
>>>
>>> I don't know why the couchdb is trying to encode the binary post  
>>> request as
>>> json.
>>>
>>> Greetings,
>>>
>>> Lennart
>>>
>>> On Nov 10, 2009, at 11:26 PM, Paul Davis wrote:
>>>
>>>> On Tue, Nov 10, 2009 at 5:22 PM, Lennart Melzer <l.melzer@tu- 
>>>> bs.de> wrote:
>>>>>
>>>>> I just looked through the error logs and the code a bit more. I  
>>>>> think
>>>>> this
>>>>> passage describes best what seems to happen (though I don't know  
>>>>> why)
>>>>>
>>>>> So the stracktrace shows that the mochiweb passes the request to
>>>>> couch_httpd:handle_request where it seems to try to call send_json
>>>>> (whysoever). It all fails when it tries to encode the binary  
>>>>> data as a
>>>>> json
>>>>> string. This all seems to happen before couch_httpd_external.erl  
>>>>> stuff
>>>>> gets
>>>>> called, which might be able to handle binary data correctly.
>>>>>
>>>>> Here's the log snippet:
>>>>> crasher:
>>>>>  initial call: mochiweb_socket_server:acceptor_loop/1
>>>>>  pid: <0.1059.0>
>>>>>  registered_name: []
>>>>>  exception exit: {ucs,{bad_utf8_character_code}}
>>>>>    in function  xmerl_ucs:from_utf8/1
>>>>>    in call from mochijson2:json_encode_string/2
>>>>>    in call from mochijson2:'-json_encode_proplist/2-fun-0-'/3
>>>>>    in call from lists:foldl/3
>>>>>    in call from mochijson2:json_encode_proplist/2
>>>>>    in call from couch_httpd:send_json/4
>>>>>    in call from couch_httpd:handle_request/5
>>>>>    in call from mochiweb_http:headers/5
>>>>>
>>>>> Anyone out that knows more about the internals of couchdb than I  
>>>>> do? I
>>>>> would
>>>>> be grateful for any advice (gotta learn some erlang programming  
>>>>> soonish).
>>>>>
>>>>> Greetings,
>>>>>
>>>>> Lennart
>>>>>
>>>>>
>>>>> On Nov 10, 2009, at 12:13 AM, Paul Davis wrote:
>>>>>
>>>>>> On Mon, Nov 9, 2009 at 12:15 PM, Lennart Melzer <l.melzer@tu-bs.de

>>>>>> >
>>>>>> wrote:
>>>>>>>
>>>>>>> I am running Revision 832477 of the couchdb trunk
>>>>>>> On Nov 9, 2009, at 6:09 PM, Paul Davis wrote:
>>>>>>>
>>>>>>>> On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l.melzer@tu-bs.de

>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Right now, posting or putting binary data to an external
 
>>>>>>>>> handler
>>>>>>>>> fails,
>>>>>>>>> since the data is parsed as a UTF-8 String and not treated
 
>>>>>>>>> as binary.
>>>>>>>>> The
>>>>>>>>> external handler never gets to do its work, so this might
be  
>>>>>>>>> an issue
>>>>>>>>> with
>>>>>>>>> the handling of data sent to an external handler. Am
I  
>>>>>>>>> misusing the
>>>>>>>>> external-handler interface?
>>>>>>>>>
>>>>>>>>> Error snippet:
>>>>>>>>> [error] [<0.95.0>] {error_report,<0.30.0>,
>>>>>>>>> {<0.95.0>,std_error,
>>>>>>>>> {mochiweb_socket_server,235,
>>>>>>>>>     {child_error,{ucs,{bad_utf8_character_code}}}}}}
>>>>>>>>>
>>>>>>>>> Greetings,
>>>>>>>>>
>>>>>>>>> Lennart
>>>>>>>>>
>>>>>>>>
>>>>>>>> Lennart,
>>>>>>>>
>>>>>>>> What version of CouchDB are you on? I seem to remember a
 
>>>>>>>> patch for
>>>>>>>> this going into trunk at some point. Not sure if that was
pre/ 
>>>>>>>> post
>>>>>>>> 0.10 though.
>>>>>>>>
>>>>>>>> Paul Davis
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Lennart,
>>>>>>
>>>>>> Hmm, looking through the svn log I'm not seeing a commit that  
>>>>>> mentions
>>>>>> body or posts. I swear I remember something like that coming  
>>>>>> through
>>>>>> at one point. Benoit, didn't you have something to do with that?
>>>>>>
>>>>>> Paul Davis
>>>>>
>>>>>
>>>>
>>>> Lennart,
>>>>
>>>> That stack trace suggests that you're trying to return a JSON  
>>>> response
>>>> that is not valid JSON. If your external is trying to return  
>>>> arbitrary
>>>> binary data in the _external response object you'll need to make  
>>>> sure
>>>> and use the body member.
>>>>
>>>> Paul Davis
>>>
>>>
>>
>> Lennart,
>>
>> What are you returning from your _external handler? You should be
>> returning something like:
>>
>> {
>>   "code": 200,
>>   "body": "arbitrary string of data and some stuff",
>>   "headers": {"Content-Type": "application/my-stuff"}
>> }
>>
>> I'd have to remind myself on the exact rules, but returning {"code":
>> 200, "json": "stuff"} will send "stuff" through the JSON serializer,
>> which means it has to conform to unicode escaping rules that
>> mochijson2 exposes via xmerl:ucs16_to_char or whatever.
>>
>> Perhaps you can log the response you're sending and paste a copy of  
>> that?
>>
>> HTH,
>> Paul Davis
>


Mime
View raw message