couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Newson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (COUCHDB-864) multipart/related PUT's always close the connection.
Date Thu, 26 Aug 2010 21:05:53 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903053#action_12903053
] 

Robert Newson commented on COUCHDB-864:
---------------------------------------

My concern is that only the process dictionary one-liner was necessary to allow many (thousands
of) multipart/related PUT's on a stable connection. It still works with Filipe's patch, of
course, but I think we should understand why it appears to work without.

It was revealed that latest mochiweb (the one with ssl) doesn't do unrecv any more, which
was the previous way pipelining was thought to work.

> multipart/related PUT's always close the connection.
> ----------------------------------------------------
>
>                 Key: COUCHDB-864
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-864
>             Project: CouchDB
>          Issue Type: Bug
>          Components: Database Core
>            Reporter: Robert Newson
>         Attachments: chunked.erl, mp_doc_put_http_pipeline.patch, mp_pipeline.patch
>
>
> I noticed that mochiweb always closes the connection when doing a multipart/related PUT
(to insert the JSON document and accompanying attachments in one call). Ultimately it's because
we call recv(0) and not recv_body, thus consuming more data than we actually process. Mochiweb
notices that there is unread data on the socket and closes the connection.
> This impacts replication with attachments, as I believe they go through this code path
(and, thus, are forever reconnecting).
> The code below demonstrates a fix for this issue but isn't good enough for trunk. Adam
provided the important process dictionary fix.
> ---
>  src/couchdb/couch_doc.erl      |    1 +
>  src/couchdb/couch_httpd_db.erl |   13 +++++++++----
>  2 files changed, 10 insertions(+), 4 deletions(-)
> diff --git a/src/couchdb/couch_doc.erl b/src/couchdb/couch_doc.erl
> index 5009f8f..f8c874b 100644
> --- a/src/couchdb/couch_doc.erl
> +++ b/src/couchdb/couch_doc.erl
> @@ -455,6 +455,7 @@ doc_from_multi_part_stream(ContentType, DataFun) ->
>      Parser ! {get_doc_bytes, self()},
>      receive 
>      {doc_bytes, DocBytes} ->
> +        erlang:put(mochiweb_request_recv, true),
>          Doc = from_json_obj(?JSON_DECODE(DocBytes)),
>          % go through the attachments looking for 'follows' in the data,
>          % replace with function that reads the data from MIME stream.
> diff --git a/src/couchdb/couch_httpd_db.erl b/src/couchdb/couch_httpd_db.erl
> index b0fbe8d..eff7d67 100644
> --- a/src/couchdb/couch_httpd_db.erl
> +++ b/src/couchdb/couch_httpd_db.erl
> @@ -651,12 +651,13 @@ db_doc_req(#httpd{method='PUT'}=Req, Db, DocId) ->
>      } = parse_doc_query(Req),
>      couch_doc:validate_docid(DocId),
>      
> +    Len = couch_httpd:header_value(Req,"Content-Length"),
>      Loc = absolute_uri(Req, "/" ++ ?b2l(Db#db.name) ++ "/" ++ ?b2l(DocId)),
>      RespHeaders = [{"Location", Loc}],
>      case couch_util:to_list(couch_httpd:header_value(Req, "Content-Type")) of
>      ("multipart/related;" ++ _) = ContentType ->
>          {ok, Doc0} = couch_doc:doc_from_multi_part_stream(ContentType,
> -                fun() -> receive_request_data(Req) end),
> +                fun() -> receive_request_data(Req, Len) end),
>          Doc = couch_doc_from_req(Req, DocId, Doc0),
>          update_doc(Req, Db, DocId, Doc, RespHeaders, UpdateType);
>      _Else ->
> @@ -775,9 +776,13 @@ send_docs_multipart(Req, Results, Options) ->
>      couch_httpd:send_chunk(Resp, <<"--">>),
>      couch_httpd:last_chunk(Resp).
>  
> -receive_request_data(Req) ->
> -    {couch_httpd:recv(Req, 0), fun() -> receive_request_data(Req) end}.
> -    
> +receive_request_data(Req, undefined) ->
> +    receive_request_data(Req, 0);
> +receive_request_data(Req, Len) when is_list(Len)->
> +    Remaining = list_to_integer(Len),
> +    Bin = couch_httpd:recv(Req, Remaining),
> +    {Bin, fun() -> receive_request_data(Req, Remaining - iolist_size(Bin)) end}.
> +
>  update_doc_result_to_json({{Id, Rev}, Error}) ->
>          {_Code, Err, Msg} = couch_httpd:error_info(Error),
>          {[{id, Id}, {rev, couch_doc:rev_to_str(Rev)},
> -- 
> 1.7.2.2
> Umbra

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message