On Oct 15, 2008, at 16:29 , Patrick Antivackis wrote:

> Hello,
> Before I fill a JIRA bug and propose a patch to fix it, i would like  
> to
> validate this issue with you.
>
> Issue : Cannot upload a file which size > 64M using PUT method
> How to reproduce :
> Create a database test
> Create a 64M file :
> dd count=131072 if=/dev/zero of=dummy64M
>
> upload this file using CURL :
> curl -H "Content-Type: application/octet-stream" -T dummy64M
> localhost:5984/test/dummy64M/ -v
> it works :
>
> [patrick@localhost ~]$ curl -H "Content-Type: application/octet- 
> stream" -T
> dummy64M localhost:5984/try/dummy64M/ -v
> * About to connect() to localhost port 5984 (#0)
> *   Trying 127.0.0.1... connected
> * Connected to localhost (127.0.0.1) port 5984 (#0)
>> PUT /try/dummy64M/dummy64M HTTP/1.1
>> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/ 
>> 0.9.8e
> zlib/1.2.3 libidn/1.0
>> Host: localhost:5984
>> Accept: */*
>> Content-Type: application/octet-stream
>> Content-Length: 67108864
>> Expect: 100-continue
>>
> < HTTP/1.1 100 Continue
>  % Total    % Received % Xferd  Average Speed   Time    Time     Time
> Current
>                                 Dload  Upload   Total   Spent    Left
> Speed
>  0 64.0M    0     0    0     0      0      0 --:--:-- --:--:--  
> --:--:--
> 0} [data not shown]
> 100 64.0M    0     0  100 64.0M      0  16.3M  0:00:03  0:00:03  
> --:--:--
> 16.3M< HTTP/1.1 201 Created
> < Server: CouchDB/0.9.0a-incubating (Erlang OTP/R12B)
> < Date: Wed, 15 Oct 2008 14:12:00 GMT
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 46
> < Cache-Control: must-revalidate
> <
> { [data not shown]
> 100 64.0M    0    46  100 64.0M      9  13.4M  0:00:04  0:00:04  
> --:--:--
> 13.0M* Connection #0 to host localhost left intact
>
> * Closing connection #0
> {"ok":true,"id":"dummy64M","rev":"1158604681"}
>
> now create a 128M file :
> dd count=262144 if=/dev/zero of=dummy128M
>
> upload this file using CURL :
> curl -H "Content-Type: application/octet-stream" -T dummy128M
> localhost:5984/test/dummy128M/ -v
> it breaks :
>
> [patrick@localhost ~]$ curl -H "Content-Type:
> application/application/octet-stream" -T dummy128M
> localhost:5984/try/dummy128M/ -v
> * About to connect() to localhost port 5984 (#0)
> *   Trying 127.0.0.1... connected
> * Connected to localhost (127.0.0.1) port 5984 (#0)
>> PUT /try/dummy128M/dummy128M HTTP/1.1
>> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/ 
>> 0.9.8e
> zlib/1.2.3 libidn/1.0
>> Host: localhost:5984
>> Accept: */*
>> Content-Type: application/application/octet-stream
>> Content-Length: 134217728
>> Expect: 100-continue
>>
> < HTTP/1.1 100 Continue
>  % Total    % Received % Xferd  Average Speed   Time    Time     Time
> Current
>                                 Dload  Upload   Total   Spent    Left
> Speed
>  0  128M    0     0    0     0      0      0 --:--:-- --:--:--  
> --:--:--
> 0} [data not shown]
> * select/poll returned error
>  0  128M    0     0    0 16384      0  2177k  0:01:00 --:--:--   
> 0:01:00
> 5333k* Closing connection #0
>
> curl: (55) select/poll returned error
>
> There is no error in couchDB logs.
>
> After debugging the code, I found that the problem occurs in
> mochiweb_request line 138 with the gen_tcp:recv  call :
>
> recv(Length, Timeout) ->
>    case gen_tcp:recv(Socket, Length, Timeout) of
>        {ok, Data} ->
>            put(?SAVE_RECV, true),
>            Data;
>        _ ->
>            exit(normal)
>    end.
>
>
> The answer catched by the case  _  is in fact  {error, enomem}
>
> There is no error in couchdb as the exit is "normal" !!!
>
> For what i read (see answers from
> http://www.google.fr/search?q=16M+gen_tcp%3Arecv), if Length is too  
> big
> (some says 16M, for me it's 64M) gen_tcp:recv sends back an enomem  
> error.
>
> I made a patch for this in order to call many time gen_tcp:recv with  
> a 16M
> Length, and then concat the binary answers (almost like in the  
> read_chunk
> method of the mochiweb_multipart).
>
> Anyway, my questions are :
> Is it really a bug or is it a way to force curl to use chunk with a  
> PUT
> method (not found any but...)

Set the Transfer-Coding header to "chunked" to force curl to chunk the  
request.
I successfully uploaded several gigabytes of data using this.

Not sure about the fix.

Cheers
Jan
--


> Should the corrections occurs here or should it be given to mochiweb  
> even if
> they seem to use PUT only for small upload files.
>
> Thank you for your answers.