incubator-couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Antivackis" <patrick.antivac...@gmail.com>
Subject Issue : upload an attachment using a PUT method
Date Wed, 15 Oct 2008 14:29:07 GMT
Hello,
Before I fill a JIRA bug and propose a patch to fix it, i would like to
validate this issue with you.

Issue : Cannot upload a file which size > 64M using PUT method
How to reproduce :
Create a database test
Create a 64M file :
dd count=131072 if=/dev/zero of=dummy64M

upload this file using CURL :
curl -H "Content-Type: application/octet-stream" -T dummy64M
localhost:5984/test/dummy64M/ -v
it works :

[patrick@localhost ~]$ curl -H "Content-Type: application/octet-stream" -T
dummy64M localhost:5984/try/dummy64M/ -v
* About to connect() to localhost port 5984 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 5984 (#0)
> PUT /try/dummy64M/dummy64M HTTP/1.1
> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3 libidn/1.0
> Host: localhost:5984
> Accept: */*
> Content-Type: application/octet-stream
> Content-Length: 67108864
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
Current
                                 Dload  Upload   Total   Spent    Left
Speed
  0 64.0M    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--
0} [data not shown]
100 64.0M    0     0  100 64.0M      0  16.3M  0:00:03  0:00:03 --:--:--
16.3M< HTTP/1.1 201 Created
< Server: CouchDB/0.9.0a-incubating (Erlang OTP/R12B)
< Date: Wed, 15 Oct 2008 14:12:00 GMT
< Content-Type: text/plain;charset=utf-8
< Content-Length: 46
< Cache-Control: must-revalidate
<
{ [data not shown]
100 64.0M    0    46  100 64.0M      9  13.4M  0:00:04  0:00:04 --:--:--
13.0M* Connection #0 to host localhost left intact

* Closing connection #0
{"ok":true,"id":"dummy64M","rev":"1158604681"}

now create a 128M file :
dd count=262144 if=/dev/zero of=dummy128M

upload this file using CURL :
curl -H "Content-Type: application/octet-stream" -T dummy128M
localhost:5984/test/dummy128M/ -v
it breaks :

[patrick@localhost ~]$ curl -H "Content-Type:
application/application/octet-stream" -T dummy128M
localhost:5984/try/dummy128M/ -v
* About to connect() to localhost port 5984 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 5984 (#0)
> PUT /try/dummy128M/dummy128M HTTP/1.1
> User-Agent: curl/7.16.4 (i686-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3 libidn/1.0
> Host: localhost:5984
> Accept: */*
> Content-Type: application/application/octet-stream
> Content-Length: 134217728
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time
Current
                                 Dload  Upload   Total   Spent    Left
Speed
  0  128M    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--
0} [data not shown]
* select/poll returned error
  0  128M    0     0    0 16384      0  2177k  0:01:00 --:--:--  0:01:00
5333k* Closing connection #0

curl: (55) select/poll returned error

There is no error in couchDB logs.

After debugging the code, I found that the problem occurs in
mochiweb_request line 138 with the gen_tcp:recv  call :

recv(Length, Timeout) ->
    case gen_tcp:recv(Socket, Length, Timeout) of
        {ok, Data} ->
            put(?SAVE_RECV, true),
            Data;
        _ ->
            exit(normal)
    end.


The answer catched by the case  _  is in fact  {error, enomem}

There is no error in couchdb as the exit is "normal" !!!

For what i read (see answers from
http://www.google.fr/search?q=16M+gen_tcp%3Arecv), if Length is too big
(some says 16M, for me it's 64M) gen_tcp:recv sends back an enomem error.

I made a patch for this in order to call many time gen_tcp:recv with a 16M
Length, and then concat the binary answers (almost like in the read_chunk
method of the mochiweb_multipart).

Anyway, my questions are :
Is it really a bug or is it a way to force curl to use chunk with a PUT
method (not found any but...)
Should the corrections occurs here or should it be given to mochiweb even if
they seem to use PUT only for small upload files.

Thank you for your answers.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message