Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Message-ID: <478533521.112441267842207300.JavaMail.jira@brutus.apache.org>
Date: Sat, 6 Mar 2010 02:23:27 +0000 (UTC)
From: "Filipe Manana (JIRA)" <jira@apache.org>
To: dev@couchdb.apache.org
Subject: [jira] Commented: (COUCHDB-679) Pull replication no longer works
 when a source doc has a large attachment
In-Reply-To: <1394900866.35821267622433213.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/COUCHDB-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842157#action_12842157 ] 

Filipe Manana commented on COUCHDB-679:
---------------------------------------

By the way, after ticket 639, this test now takes about 5 secs of running time in my machine. Before it took about 11 seconds.
This is due to the fact that base64 encoding/decoding is no longer needed for the push replication.

> Pull replication no longer works when a source doc has a large attachment
> -------------------------------------------------------------------------
>
>                 Key: COUCHDB-679
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-679
>             Project: CouchDB
>          Issue Type: Bug
>          Components: Replication
>         Environment: trunk and 0.11
>            Reporter: Filipe Manana
>            Priority: Blocker
>             Fix For: 0.11
>
>         Attachments: CouchDB-679-etap_test-trunk-2.patch, CouchDB-679-etap_test-trunk-3.patch, CouchDB-679-etap_test-trunk.patch
>
>
> After rev r916868  (fix for CouchDB-597), doing a pull replication of a doc that has a large attachment no longer works.
> The problem is in couch_rep_att.erl:
> convert_stub(#att{data=stub, name=Name} = Attachment,
> 	     {#http_db{} = Db, Id, Rev}) ->
>     {Pos, [RevId|_]} = Rev,
>     Request = Db#http_db{
>         resource = lists:flatten([couch_util:url_encode(Id), "/",
>             couch_util:url_encode(Name)]),
>         qs = [{rev, couch_doc:rev_to_str({Pos,RevId})}]
>     },
>     Ref = make_ref(),
>     RcvFun = fun() ->
>                  Bin = attachment_receiver(Ref, Request),
>                  cleanup(),
>                  Bin
>              end,
>     Attachment#att{data=RcvFun}.
> The cleanup/0 function can not be called there, since when the attachment is received in multiple chunks, after receiving the first chunk the subsequent ones are silently discarded by cleanup/0:
> cleanup() ->
>  	receive
>  	{ibrowse_async_response, _, _} ->
>  	   %% TODO maybe log, didn't expect to have data here
>  	   cleanup();
>  	{ibrowse_async_response_end, _} ->
>  	    cleanup();
>  	{ibrowse_async_headers, _, _, _} ->
>  	    cleanup()
>  	after 0 ->
>  	    erase(),
>  	    ok
>  	end. 
> If you look into couch_db.erl, you'll see that the attachment receiver function maybe called multiple times:
> flush_att(Fd, #att{data=Fun,att_len=AttLen}=Att) when is_function(Fun) ->
>     with_stream(Fd, Att, fun(OutputStream) ->
>         write_streamed_attachment(OutputStream, Fun, AttLen)
>     end).
> write_streamed_attachment(_Stream, _F, 0) ->
>     ok;
> write_streamed_attachment(Stream, F, LenLeft) ->
>     Bin = F(),
>     ok = couch_stream:write(Stream, Bin),
>     write_streamed_attachment(Stream, F, LenLeft - size(Bin)).
> This is serious :(
> However, removing that call to cleanup/0 may cause the return of CouchDB-597, therefore I don't supply a patch here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.