couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Filipe Manana (JIRA)" <j...@apache.org>
Subject [jira] Created: (COUCHDB-679) Pull replication no longer works when a source doc has a large attachment
Date Wed, 03 Mar 2010 13:20:33 GMT
Pull replication no longer works when a source doc has a large attachment
-------------------------------------------------------------------------

                 Key: COUCHDB-679
                 URL: https://issues.apache.org/jira/browse/COUCHDB-679
             Project: CouchDB
          Issue Type: Bug
          Components: Replication
         Environment: trunk and 0.11
            Reporter: Filipe Manana
            Priority: Blocker
             Fix For: 0.11


After rev r916868  (fix for CouchDB-597), doing a pull replication of a doc that has a large
attachment no longer works.

The problem is in couch_rep_att.erl:

convert_stub(#att{data=stub, name=Name} = Attachment,
	     {#http_db{} = Db, Id, Rev}) ->
    {Pos, [RevId|_]} = Rev,
    Request = Db#http_db{
        resource = lists:flatten([couch_util:url_encode(Id), "/",
            couch_util:url_encode(Name)]),
        qs = [{rev, couch_doc:rev_to_str({Pos,RevId})}]
    },
    Ref = make_ref(),
    RcvFun = fun() ->
                 Bin = attachment_receiver(Ref, Request),
                 cleanup(),
                 Bin
             end,
    Attachment#att{data=RcvFun}.

The cleanup/0 function can not be called there, since when the attachment is received in multiple
chunks, after receiving the first chunk the subsequent ones are silently discarded by cleanup/0:

cleanup() ->
 	receive
 	{ibrowse_async_response, _, _} ->
 	   %% TODO maybe log, didn't expect to have data here
 	   cleanup();
 	{ibrowse_async_response_end, _} ->
 	    cleanup();
 	{ibrowse_async_headers, _, _, _} ->
 	    cleanup()
 	after 0 ->
 	    erase(),
 	    ok
 	end. 

If you look into couch_db.erl, you'll see that the attachment receiver function maybe called
multiple times:

flush_att(Fd, #att{data=Fun,att_len=AttLen}=Att) when is_function(Fun) ->
    with_stream(Fd, Att, fun(OutputStream) ->
        write_streamed_attachment(OutputStream, Fun, AttLen)
    end).

write_streamed_attachment(_Stream, _F, 0) ->
    ok;
write_streamed_attachment(Stream, F, LenLeft) ->
    Bin = F(),
    ok = couch_stream:write(Stream, Bin),
    write_streamed_attachment(Stream, F, LenLeft - size(Bin)).

This is serious :(

However, removing that call to cleanup/0 may cause the return of CouchDB-597, therefore I
don't supply a patch here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message