couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Newson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (COUCHDB-864) multipart/related PUT's always close the connection.
Date Fri, 20 Aug 2010 18:14:16 GMT

    [ https://issues.apache.org/jira/browse/COUCHDB-864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900809#action_12900809
] 

Robert Newson commented on COUCHDB-864:
---------------------------------------

JIRA is probably not the easiest way to convey the gist of the discussion Adam and I had today.

The first thing to clarify is that there's no bug as such, just a (significant) inefficiency.
Mochiweb closes the request and the http client works correctly when that happens.

I discovered the problem as I'm doing performance testing. I'm using nodeload (a node.js-based
http benchmarking kit). I tell it to keep the connections alive. This works fine in my original
test where I upload standalone attachments. When I switched to uploading multipart/related,
so that I can set json metadata as well, it blew up with 'connection reset by peer'. Assuming,
as one does, that it was a node.js bug, I tried the same upload in other ways and they all
behaved the same way.

CouchDB should be able to keep the connections alive for multiple requests in most cases.
It does so, as far as I can tell, in every case exception multipart/related PUT's. Having
read the related code, and discussed it with Adam on IRC, it's clear why.

I'll post a further patch later on when I've had more time to tidy and test.

> multipart/related PUT's always close the connection.
> ----------------------------------------------------
>
>                 Key: COUCHDB-864
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-864
>             Project: CouchDB
>          Issue Type: Bug
>          Components: Database Core
>            Reporter: Robert Newson
>
> I noticed that mochiweb always closes the connection when doing a multipart/related PUT
(to insert the JSON document and accompanying attachments in one call). Ultimately it's because
we call recv(0) and not recv_body, thus consuming more data than we actually process. Mochiweb
notices that there is unread data on the socket and closes the connection.
> This impacts replication with attachments, as I believe they go through this code path
(and, thus, are forever reconnecting).
> The code below demonstrates a fix for this issue but isn't good enough for trunk. Adam
provided the important process dictionary fix.
> ---
>  src/couchdb/couch_doc.erl      |    1 +
>  src/couchdb/couch_httpd_db.erl |   13 +++++++++----
>  2 files changed, 10 insertions(+), 4 deletions(-)
> diff --git a/src/couchdb/couch_doc.erl b/src/couchdb/couch_doc.erl
> index 5009f8f..f8c874b 100644
> --- a/src/couchdb/couch_doc.erl
> +++ b/src/couchdb/couch_doc.erl
> @@ -455,6 +455,7 @@ doc_from_multi_part_stream(ContentType, DataFun) ->
>      Parser ! {get_doc_bytes, self()},
>      receive 
>      {doc_bytes, DocBytes} ->
> +        erlang:put(mochiweb_request_recv, true),
>          Doc = from_json_obj(?JSON_DECODE(DocBytes)),
>          % go through the attachments looking for 'follows' in the data,
>          % replace with function that reads the data from MIME stream.
> diff --git a/src/couchdb/couch_httpd_db.erl b/src/couchdb/couch_httpd_db.erl
> index b0fbe8d..eff7d67 100644
> --- a/src/couchdb/couch_httpd_db.erl
> +++ b/src/couchdb/couch_httpd_db.erl
> @@ -651,12 +651,13 @@ db_doc_req(#httpd{method='PUT'}=Req, Db, DocId) ->
>      } = parse_doc_query(Req),
>      couch_doc:validate_docid(DocId),
>      
> +    Len = couch_httpd:header_value(Req,"Content-Length"),
>      Loc = absolute_uri(Req, "/" ++ ?b2l(Db#db.name) ++ "/" ++ ?b2l(DocId)),
>      RespHeaders = [{"Location", Loc}],
>      case couch_util:to_list(couch_httpd:header_value(Req, "Content-Type")) of
>      ("multipart/related;" ++ _) = ContentType ->
>          {ok, Doc0} = couch_doc:doc_from_multi_part_stream(ContentType,
> -                fun() -> receive_request_data(Req) end),
> +                fun() -> receive_request_data(Req, Len) end),
>          Doc = couch_doc_from_req(Req, DocId, Doc0),
>          update_doc(Req, Db, DocId, Doc, RespHeaders, UpdateType);
>      _Else ->
> @@ -775,9 +776,13 @@ send_docs_multipart(Req, Results, Options) ->
>      couch_httpd:send_chunk(Resp, <<"--">>),
>      couch_httpd:last_chunk(Resp).
>  
> -receive_request_data(Req) ->
> -    {couch_httpd:recv(Req, 0), fun() -> receive_request_data(Req) end}.
> -    
> +receive_request_data(Req, undefined) ->
> +    receive_request_data(Req, 0);
> +receive_request_data(Req, Len) when is_list(Len)->
> +    Remaining = list_to_integer(Len),
> +    Bin = couch_httpd:recv(Req, Remaining),
> +    {Bin, fun() -> receive_request_data(Req, Remaining - iolist_size(Bin)) end}.
> +
>  update_doc_result_to_json({{Id, Rev}, Error}) ->
>          {_Code, Err, Msg} = couch_httpd:error_info(Error),
>          {[{id, Id}, {rev, couch_doc:rev_to_str(Rev)},
> -- 
> 1.7.2.2
> Umbra

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message