Return-Path: X-Original-To: apmail-couchdb-dev-archive@www.apache.org Delivered-To: apmail-couchdb-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0BE43D834 for ; Thu, 11 Oct 2012 16:44:22 +0000 (UTC) Received: (qmail 42704 invoked by uid 500); 11 Oct 2012 16:44:21 -0000 Delivered-To: apmail-couchdb-dev-archive@couchdb.apache.org Received: (qmail 42678 invoked by uid 500); 11 Oct 2012 16:44:21 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 42670 invoked by uid 99); 11 Oct 2012 16:44:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Oct 2012 16:44:21 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert.newson@gmail.com designates 209.85.212.52 as permitted sender) Received: from [209.85.212.52] (HELO mail-vb0-f52.google.com) (209.85.212.52) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Oct 2012 16:44:15 +0000 Received: by mail-vb0-f52.google.com with SMTP id k17so1909728vbj.11 for ; Thu, 11 Oct 2012 09:43:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:from:in-reply-to:mime-version:date:message-id:subject:to :content-type:content-transfer-encoding; bh=8brtHaqZy9x41A+UexUMEnBmam3wCT9wjzufwt5OAHc=; b=Hgx/gtafrCLN8BOq5OfANRcpf4ZmlfRz/bfgwyS8JMWehrKp/a0r+e+X8Ns8lgYMd0 DEj14td3qhrlDENiewVaMN8WvafApDkGPntbv+/bwWh4gFrgMlDLxKjtaNfwgQvtO2kt q8EgikehLRYBHZrGrpTk2wENfQQe3Ly3scoRltXJITSwNFd+OEsYGPZUA9O1sTDYu+LS M06mmklAOykmGXZd+VeXVVF3bsCN5VDRsJTB08P5Vh3MrDCxP9io9DcgZ26cWwI2MWwz G5qPCpZcGFNmm86kGzKEXvtO6MT0Opsf4lP3lLUTzVRPZ6Jda1uJaAvE1f4XIXRi3B3N 1iEA== Received: by 10.221.11.71 with SMTP id pd7mr817613vcb.45.1349973833997; Thu, 11 Oct 2012 09:43:53 -0700 (PDT) References: <2EC65984-7E61-4523-8CE0-372BCB898AE7@spy.net> <4A889776-299A-4F57-92A3-09CCC6AE6792@spy.net> <3E55AC16-BBD6-4CBA-BB7F-F5AC2FCE566F@apache.org> <-8927617846284403981@unknownmsgid> <16A04644-43C0-4F37-8960-79F8F8DB9C7C@apache.org> <7159244169061905707@unknownmsgid> <658299EE-FC39-4D43-B57C-64CCDB19DF06@dionne-associates.com> <3805065515884149073@unknownmsgid> From: Robert Newson In-Reply-To: Mime-Version: 1.0 (1.0) Date: Thu, 11 Oct 2012 17:43:50 +0100 Message-ID: <2530680098764232101@unknownmsgid> Subject: Re: replication problems To: "dev@couchdb.apache.org" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Well, sure, if source or target are remote, the replication_id function will have to make an http call to get the uuid. We can skip that if the port is outside the dynamic range, so, mostly we only ever do these extra calls when needed, the upside of a stable checkpoint outweighing the 1 or 2 http calls when starting a replication. Easier to describe this as code, I think, than prose. Seems simple and easy= . Sent from the ocean floor On 11 Oct 2012, at 17:14, Bob Dionne wrote: > > On Oct 11, 2012, at 10:27 AM, Robert Newson wro= te: > >> I figured we'd set the same replicator uuid for all nodes of a >> cluster, same as for cookie auth. Removing host:port and the >> inet:gethostname bits will be nice. We'll still find gen 2 >> checkpoints, replacing them with gen 3. > > Yes, I think that works. There still needs to be a way to surface that fo= r both source and target, in the case the replication is mediated on a thir= d machine. Overloading the semantics of the welcome message seems ok > > > > >> Of course, it's possible all >> this relaxing by the pool stuff has damaged my brain. >> >> Sent from the ocean floor >> >> On 11 Oct 2012, at 14:59, Bob Dionne wrot= e: >> >>> Incorporating a unique id from the source and target seems like a good = way to go but I'm wondering if an id from an ini file will >>> work in the clustered BigCouch case. Would an API level request work be= tter? Something the replicator would interrogate >>> for both the source and the target. >>> >>> >>> On Oct 11, 2012, at 5:42 AM, Robert Newson wr= ote: >>> >>>> I'll note here that the attached patch is wrong. It uses a single uuid >>>> from the node running replication, which might not be the source or >>>> target. Instead, the uuid of source and target must be retrieved and >>>> used instead of the host:port. Jason's suggestion to add the uuid >>>> (stored in the ini file) to the welcome message sounds really good to >>>> me. >>>> >>>> Can't attach this to the ticket today as I don't have my Jira creds. >>>> >>>> Sent from the ocean floor >>>> >>>> On 10 Oct 2012, at 21:40, Jan Lehnardt wrote: >>>> >>>>> flagged. >>>>> >>>>> On Oct 10, 2012, at 22:34 , Robert Newson w= rote: >>>>> >>>>>> Jan, >>>>>> >>>>>> Flag that as fix-for 1.3? I don't have my creds on my phone to do it= . >>>>>> >>>>>> I like the ini uuid idea best, modelled after the cookie with secret= . >>>>>> If we have the uuid, we'd omit host name as well as port, right? >>>>>> >>>>>> Sent from the ocean floor >>>>>> >>>>>> On 10 Oct 2012, at 21:12, Jan Lehnardt wrote: >>>>>> >>>>>>> Filipe tells me this is https://issues.apache.org/jira/browse/COUCH= DB-1259 >>>>>>> >>>>>>> Cheers >>>>>>> Jan >>>>>>> -- >>>>>>> >>>>>>> On Oct 4, 2012, at 02:28 , Dustin Sallings wrote: >>>>>>> >>>>>>>> >>>>>>>> I'm bringing this back up as requested. I'm currently simultaneou= sly in the "not replicating interesting things" and "has duplicate replicat= es state". I think the stuff below shows the "not replicating" stuff. >>>>>>>> >>>>>>>> Active tasks shows the other (these are based on replicator DB doc= uments (example below): >>>>>>>> >>>>>>>> [ >>>>>>>> { >>>>>>>> "checkpointed_source_seq": 2022317, >>>>>>>> "continuous": true, >>>>>>>> "doc_id": "cbstats-from-dogbowl", >>>>>>>> "doc_write_failures": 0, >>>>>>>> "docs_read": 300, >>>>>>>> "docs_written": 300, >>>>>>>> "missing_revisions_found": 300, >>>>>>>> "pid": "<0.10466.12>", >>>>>>>> "progress": 100, >>>>>>>> "replication_id": "50daecd0a29f4b7e5d102990831f3d64+continuous", >>>>>>>> "revisions_checked": 304, >>>>>>>> "source": "http://dustin:*****@single.couchbase.net/cbstats/", >>>>>>>> "source_seq": 2022317, >>>>>>>> "started_on": 1349309457, >>>>>>>> "target": "cbstats", >>>>>>>> "type": "replication", >>>>>>>> "updated_on": 1349310442 >>>>>>>> }, >>>>>>>> { >>>>>>>> "checkpointed_source_seq": 2022317, >>>>>>>> "continuous": true, >>>>>>>> "doc_id": "cbstats-from-dogbowl", >>>>>>>> "doc_write_failures": 0, >>>>>>>> "docs_read": 62, >>>>>>>> "docs_written": 62, >>>>>>>> "missing_revisions_found": 62, >>>>>>>> "pid": "<0.11019.12>", >>>>>>>> "progress": 100, >>>>>>>> "replication_id": "411e341d5aa9a3fe636cf4ea8ba71720+continuous", >>>>>>>> "revisions_checked": 304, >>>>>>>> "source": "http://dustin:*****@single.couchbase.net/cbstats/", >>>>>>>> "source_seq": 2022317, >>>>>>>> "started_on": 1349309471, >>>>>>>> "target": "cbstats", >>>>>>>> "type": "replication", >>>>>>>> "updated_on": 1349310443 >>>>>>>> }, >>>>>>>> { >>>>>>>> "checkpointed_source_seq": 107068, >>>>>>>> "continuous": true, >>>>>>>> "doc_id": "gerrit-from-prod", >>>>>>>> "doc_write_failures": 0, >>>>>>>> "docs_read": 22, >>>>>>>> "docs_written": 22, >>>>>>>> "missing_revisions_found": 22, >>>>>>>> "pid": "<0.11086.12>", >>>>>>>> "progress": 100, >>>>>>>> "replication_id": "4a21031dac0d81637a23c32bad620be9+continuous", >>>>>>>> "revisions_checked": 26, >>>>>>>> "source": "http://dustinphoto.iriscouch.com/gerrit/", >>>>>>>> "source_seq": 107068, >>>>>>>> "started_on": 1349309487, >>>>>>>> "target": "gerrit", >>>>>>>> "type": "replication", >>>>>>>> "updated_on": 1349310445 >>>>>>>> }, >>>>>>>> { >>>>>>>> "checkpointed_source_seq": 107068, >>>>>>>> "continuous": true, >>>>>>>> "doc_id": "gerrit-from-prod", >>>>>>>> "doc_write_failures": 0, >>>>>>>> "docs_read": 17, >>>>>>>> "docs_written": 17, >>>>>>>> "missing_revisions_found": 17, >>>>>>>> "pid": "<0.11107.12>", >>>>>>>> "progress": 100, >>>>>>>> "replication_id": "b4ad5d3f2e5b78670e4c8364b18000e9+continuous", >>>>>>>> "revisions_checked": 26, >>>>>>>> "source": "http://dustinphoto.iriscouch.com/gerrit/", >>>>>>>> "source_seq": 107068, >>>>>>>> "started_on": 1349309488, >>>>>>>> "target": "gerrit", >>>>>>>> "type": "replication", >>>>>>>> "updated_on": 1349310445 >>>>>>>> } >>>>>>>> ] >>>>>>>> >>>>>>>> >>>>>>>> The replicator document for the latter, for example is this: >>>>>>>> >>>>>>>> { >>>>>>>> "_id": "gerrit-from-prod", >>>>>>>> "_rev": "2235-36de10fb757581a1782dacbb26ee4809", >>>>>>>> "source": "http://dustinphoto.iriscouch.com/gerrit", >>>>>>>> "target": "gerrit", >>>>>>>> "continuous": true, >>>>>>>> "user_ctx": { >>>>>>>> "roles": [ >>>>>>>> "_admin" >>>>>>>> ] >>>>>>>> }, >>>>>>>> "_replication_state_time": "2012-10-03T17:11:27-07:00", >>>>>>>> "_replication_id": "b4ad5d3f2e5b78670e4c8364b18000e9", >>>>>>>> "_replication_state": "triggered" >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> Begin forwarded message: >>>>>>>> >>>>>>>>> From: Dustin Sallings >>>>>>>>> Subject: Re: replication problems >>>>>>>>> Date: June 15, 2012 0:10:04 PDT >>>>>>>>> To: dev@couchdb.apache.org >>>>>>>>> Reply-To: dev@couchdb.apache.org >>>>>>>>> >>>>>>>>> >>>>>>>>> On Jun 14, 2012, at 11:28 PM, Benoit Chesneau wrote: >>>>>>>>> >>>>>>>>>> Ar you using _replicate or _replicator ? Anything interresting i= n logs? >>>>>>>>> >>>>>>>>> >>>>>>>>> I'm using _replicator (wonderful feature, I just kill the DB and = everything goes back the way I want it). >>>>>>>>> >>>>>>>>> Hmm... I do think I found some stuff digging through the logs. = This is the local DB I noticed not doing its thing, although there were ton= s of errors all around this. Looks like the server got into some kind of b= ad state and sort of half-crashed. >>>>>>>>> >>>>>>>>> >>>>>>>>> [Thu, 14 Jun 2012 23:20:12 GMT] [error] [<0.133.0>] Replication `= ae601df0373da82d1b4a9ff741c8ba18+continuous` (`rpics` -> `rpics-processed`)= failed: {{timeout,{gen_server,call,[<0.213.0>,{open_ref_count,<0.4 >>>>>>>>> 42.0>}]}}, >>>>>>>>> {gen_server,call, >>>>>>>>> [couch_server, >>>>>>>>> {open,<<"rpics">>, >>>>>>>>> [{user_ctx,{user_ctx,null,[<<"_admin">>],undefined}}]}= , >>>>>>>>> infinity]}} >>>>>>>>> [Thu, 14 Jun 2012 23:20:25 GMT] [error] [<0.383.0>] ** Generic se= rver <0.383.0> terminating >>>>>>>>> ** Last message in was {'EXIT',<0.384.0>, >>>>>>>>> {{timeout, >>>>>>>>> {gen_server,call, >>>>>>>>> [<0.213.0>,{open_ref_count,<0.442.0>}]}}, >>>>>>>>> {gen_server,call, >>>>>>>>> [couch_server, >>>>>>>>> {open,<<"cbstats">>, >>>>>>>>> [{user_ctx, >>>>>>>>> {user_ctx,null,[<<"_admin">>],undefined}}, >>>>>>>>> {user_ctx, >>>>>>>>> {user_ctx,null,[<<"_admin">>],undefined}}]}, >>>>>>>>> infinity]}}} >>>>>>>>> >>>>>>>>> ** When Server state =3D=3D {state,<0.272.0>,<0.384.0>,20, >>>>>>>>> {httpdb, >>>>>>>>> "http://dustin:LOGGED_PASSWORD@single.couchbase.= net/cbstats/", >>>>>>>>> nil, >>>>>>>>> [{"Accept","application/json"}, >>>>>>>>> {"User-Agent","CouchDB/1.2.0"}], >>>>>>>>> 30000, >>>>>>>>> [{socket_options, >>>>>>>>> [{keepalive,true},{nodelay,false}]}], >>>>>>>>> 10,250,<0.273.0>,20}, >>>>>>>>> {db,<0.288.0>,<0.289.0>,nil,<<"1339637701848579">= >, >>>>>>>>> <0.290.0>,<0.286.0>,<0.367.0>, >>>>>>>>> {db_header,6,984356,0, >>>>>>>>> {860345646,{737369,975,640891414},59433736}, >>>>>>>>> {860348005,738344,42056446}, >>>>>>>>> {860352635,[],5737}, >>>>>>>>> 0,nil,nil,1000}, >>>>>>>>> 984356, >>>>>>>>> {btree,<0.286.0>, >>>>>>>>> {860345646,{737369,975,640891414},59433736}, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun,snappy}, >>>>>>>>> {btree,<0.286.0>, >>>>>>>>> {860348005,738344,42056446}, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun,snappy}, >>>>>>>>> {btree,<0.286.0>, >>>>>>>>> {860352635,[],5737}, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun,nil,snappy}, >>>>>>>>> 984356,<<"cbstats">>, >>>>>>>>> "/Volumes/terror/db/couchdb/cbstats.couch",[],[]= , >>>>>>>>> nil, >>>>>>>>> {user_ctx,null,[<<"_admin">>],undefined}, >>>>>>>>> nil,1000, >>>>>>>>> [before_header,after_header,on_file_open], >>>>>>>>> [{user_ctx, >>>>>>>>> {user_ctx,null,[<<"_admin">>],undefined}}], >>>>>>>>> snappy,nil,nil}, >>>>>>>>> [],nil,nil,nil, >>>>>>>>> {rep_stats,0,0,0,0,0}, >>>>>>>>> nil,<0.385.0>, >>>>>>>>> {batch,[],0}} >>>>>>>>> ** Reason for termination =3D=3D >>>>>>>>> ** {noproc,{gen_server,call,[<0.367.0>,{drop,<0.383.0>},infinity]= }} >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Scrolling to the beginning of the errors, I find this: >>>>>>>>> >>>>>>>>> >>>>>>>>> [Thu, 14 Jun 2012 23:15:54 GMT] [error] [<0.164.0>] Replication `= 543f76281e8d52d6ce5b51fddf0588e7+continuous` (`photo` -> `http://dustin:***= **@dustinphoto.couchone.com/photo/`) failed: source_db_down >>>>>>>>> [Thu, 14 Jun 2012 23:18:57 GMT] [info] [<0.358.0>] 127.0.0.1 - - = GET /_all_dbs 200 >>>>>>>>> [Thu, 14 Jun 2012 23:19:52 GMT] [error] [<0.289.0>] ** Generic se= rver <0.289.0> terminating >>>>>>>>> ** Last message in was {update_docs,<0.272.0>,[], >>>>>>>>> [{{doc, >>>>>>>>> <<"_local/c4cc070f896d7267e52ba012856fed4= b">>, >>>>>>>>> {0,[<<"346185">>]}, >>>>>>>>> {[{<<"session_id">>, >>>>>>>>> <<"9fb3475683d44bb1e151031dd42cc59f">>= }, >>>>>>>>> {<<"source_last_seq">>,1419004}, >>>>>>>>> {<<"replication_id_version">>,2}, >>>>>>>>> {<<"history">>, >>>>>>>>> [{[{<<"session_id">>, >>>>>>>>> <<"9fb3475683d44bb1e151031dd42cc59= f">>}, >>>>>>>>> {<<"start_time">>, >>>>>>>>> <<"Thu, 14 Jun 2012 01:35:02 GMT">= >}, >>>>>>>>> {<<"end_time">>, >>>>>>>>> <<"Thu, 14 Jun 2012 23:15:29 GMT">= >}, >>>>>>>>> {<<"start_last_seq">>,1410146}, >>>>>>>>> {<<"end_last_seq">>,1419004}, >>>>>>>>> {<<"recorded_seq">>,1419004}, >>>>>>>>> {<<"missing_checked">>,8100}, >>>>>>>>> {<<"missing_found">>,8100}, >>>>>>>>> {<<"docs_read">>,8100}, >>>>>>>>> {<<"docs_written">>,8100}, >>>>>>>>> {<<"doc_write_failures">>,0}]}, >>>>>>>>> {[{<<"session_id">>, >>>>>>>>> <<"3edd7c50327eab7ec0768451e34efa8= b">>}, >>>>>>>>> {<<"start_time">>, >>>>>>>>> <<"Tue, 12 Jun 2012 05:51:17 GMT">= >}, >>>>>>>>> {<<"end_time">>, >>>>>>>>> <<"Tue, 12 Jun 2012 13:02:37 GMT">= >}, >>>>>>>>> {<<"start_last_seq">>,1407186}, >>>>>>>>> {<<"end_last_seq">>,1410146}, >>>>>>>>> {<<"recorded_seq">>,1410146}, >>>>>>>>> {<<"missing_checked">>,2583}, >>>>>>>>> {<<"missing_found">>,2577}, >>>>>>>>> {<<"docs_read">>,2577}, >>>>>>>>> {<<"docs_written">>,2577}, >>>>>>>>> {<<"doc_write_failures">>,0}]}, >>>>>>>>> {[{<<"session_id">>, >>>>>>>>> <<"172de62044281a01b1584a9d099f42a= f">>}, >>>>>>>>> {<<"start_time">>, >>>>>>>>> <<"Mon, 11 Jun 2012 03:40:11 GMT">= >}, >>>>>>>>> {<<"end_time">>, >>>>>>>>> <<"Mon, 11 Jun 2012 15:16:24 GMT">= >}, >>>>>>>>> {<<"start_last_seq">>,1405428}, >>>>>>>>> {<<"end_last_seq">>,1407186}, >>>>>>>>> {<<"recorded_seq">>,1407186}, >>>>>>>>> {<<"missing_checked">>,1721}, >>>>>>>>> {<<"missing_found">>,1721}, >>>>>>>>> {<<"docs_read">>,1721}, >>>>>>>>> {<<"docs_written">>,1721}, >>>>>>>>> {<<"doc_write_failures">>,0}]}, >>>>>>>>> {[{<<"session_id">>, >>>>>>>>> <<"e60a126a2036c5fab00a1249101820c= 8">>}, >>>>>>>>> {<<"start_time">>, >>>>>>>>> <<"Sat, 09 Jun 2012 07:47:22 GMT">= >}, >>>>>>>>> {<<"end_time">>, >>>>>>>>> <<"Sun, 10 Jun 2012 21:16:20 GMT">= >}, >>>>>>>>> {<<"start_last_seq">>,1386289}, >>>>>>>>> {<<"end_last_seq">>,1405428}, >>>>>>>>> {<<"recorded_seq">>,1405428}, >>>>>>>>> {<<"missing_checked">>,16977}, >>>>>>>>> {<<"missing_found">>,16977}, >>>>>>>>> {<<"docs_read">>,16977}, >>>>>>>>> {<<"docs_written">>,16977}, >>>>>>>>> {<<"doc_write_failures">>,0}]}, >>>>>>>>> {[{<<"session_id">>, >>>>>>>>> <<"ef3e4333d340dcf73ddfa3fe8c72004= 2">>}, >>>>>>>>> {<<"start_time">>, >>>>>>>>> <<"Mon, 04 Jun 2012 02:39:44 GMT">= >}, >>>>>>>>> {<<"end_time">>, >>>>>>>>> <<"Mon, 04 Jun 2012 12:35:50 GMT">= >}, >>>>>>>>> {<<"start_last_seq">>,1384738}, >>>>>>>>> {<<"end_last_seq">>,1386289}, >>>>>>>>> {<<"recorded_seq">>,1386289}, >>>>>>>>> {<<"missing_checked">>,1551}, >>>>>>>>> {<<"missing_found">>,1550}, >>>>>>>>> {<<"docs_read">>,1550}, >>>>>>>>> {<<"docs_written">>,1550}, >>>>>>>>> {<<"doc_write_failures">>,0}]}, >>>>>>>>> {[{<<"session_id">>, >>>>>>>>> <<"d5123a3caf462794aaf5a47be1bb3b6= e">>}, >>>>>>>>> {<<"start_time">>, >>>>>>>>> <<"Wed, 30 May 2012 20:41:43 GMT">= >}, >>>>>>>>> {<<"end_time">>, >>>>>>>>> <<"Mon, 04 Jun 2012 02:37:33 GMT">= >}, >>>>>>>>> {<<"start_last_seq">>,1372404}, >>>>>>>>> {<<"end_last_seq">>,1384738}, >>>>>>>>> {<<"recorded_seq">>,1384738}, >>>>>>>>> {<<"missing_checked">>,12334}, >>>>>>>>> {<<"missing_found">>,12333}, >>>>>>>>> {<<"docs_read">>,12333}, >>>>>>>>> {<<"docs_written">>,12333}, >>>>>>>>> {<<"doc_write_failures">>,0}]}, >>>>>>>>> {[{<<"session_id">>, >>>>>>>>> <<"52a16e8832f70dc094f6fff5e9b7d75= b">>}, >>>>>>>>> {<<"start_time">>, >>>>>>>>> <<"Sun, 27 May 2012 23:36:41 GMT">= >}, >>>>>>>>> {<<"end_time">>, >>>>>>>>> <<"Wed, 30 May 2012 20:40:14 GMT">= >}, >>>>>>>>> {<<"start_last_seq">>,1361049}, >>>>>>>>> {<<"end_last_seq">>,1372404}, >>>>>>>>> {<<"recorded_seq">>,1372404}, >>>>>>>>> {<<"missing_checked">>,11355}, >>>>>>>>> {<<"missing_found">>,11355}, >>>>>>>>> {<<"docs_read">>,11355}, >>>>>>>>> {<<"docs_written">>,11355}, >>>>>>>>> {<<"doc_write_failures">>,0}]}, >>>>>>>>> [...lots of these...] >>>>>>>>> >>>>>>>>> [],false,[]}, >>>>>>>>> #Ref<0.0.15.159973>}], >>>>>>>>> false,false} >>>>>>>>> ** When Server state =3D=3D {db,<0.288.0>,<0.289.0>,nil,<<"133963= 7701848579">>, >>>>>>>>> <0.290.0>,<0.286.0>,<0.367.0>, >>>>>>>>> {db_header,6,992456,0, >>>>>>>>> {943280145,{744250,975,647546641},60017672= }, >>>>>>>>> {943282327,745225,42485979}, >>>>>>>>> {943267963,[],5753}, >>>>>>>>> 0,nil,nil,1000}, >>>>>>>>> 992456, >>>>>>>>> {btree,<0.286.0>, >>>>>>>>> {943280145,{744250,975,647546641},60017672= }, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun,snappy}= , >>>>>>>>> {btree,<0.286.0>, >>>>>>>>> {943282327,745225,42485979}, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun,snappy}= , >>>>>>>>> {btree,<0.286.0>, >>>>>>>>> {943267963,[],5753}, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun,nil,snappy}, >>>>>>>>> 992456,<<"cbstats">>, >>>>>>>>> "/Volumes/terror/db/couchdb/cbstats.couch",[],= [], >>>>>>>>> nil, >>>>>>>>> {user_ctx,null,[],undefined}, >>>>>>>>> nil,1000, >>>>>>>>> [before_header,after_header,on_file_open], >>>>>>>>> [{user_ctx, >>>>>>>>> {user_ctx,null,[<<"_admin">>],undefined}}= ], >>>>>>>>> snappy,nil,nil} >>>>>>>>> ** Reason for termination =3D=3D >>>>>>>>> ** {timeout, >>>>>>>>> {gen_server,call, >>>>>>>>> [<0.288.0>, >>>>>>>>> {db_updated, >>>>>>>>> {db,<0.288.0>,<0.289.0>,nil,<<"1339637701848579">>,<0.290.= 0>, >>>>>>>>> <0.286.0>,<0.367.0>, >>>>>>>>> {db_header,6,992456,0, >>>>>>>>> {943280145,{744250,975,647546641},60017672}, >>>>>>>>> {943282327,745225,42485979}, >>>>>>>>> {943267963,[],5753}, >>>>>>>>> 0,nil,nil,1000}, >>>>>>>>> 992456, >>>>>>>>> {btree,<0.286.0>, >>>>>>>>> {943280145,{744250,975,647546641},60017672}, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun,snappy}, >>>>>>>>> {btree,<0.286.0>, >>>>>>>>> {943282327,745225,42485979}, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun,snappy}, >>>>>>>>> {btree,<0.286.0>, >>>>>>>>> {943284347,[],5756}, >>>>>>>>> #Fun, >>>>>>>>> #Fun, >>>>>>>>> #Fun,nil,snappy}, >>>>>>>>> 992456,<<"cbstats">>, >>>>>>>>> "/Volumes/terror/db/couchdb/cbstats.couch",[],[],nil, >>>>>>>>> {user_ctx,null,[],undefined}, >>>>>>>>> #Ref<0.0.15.160107>,1000, >>>>>>>>> [before_header,after_header,on_file_open], >>>>>>>>> [{user_ctx,{user_ctx,null,[<<"_admin">>],undefined}}], >>>>>>>>> snappy,nil,nil}}]}} >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> dustin sallings >>>>>>>> >>>>>>>> -- >>>>>>>> dustin sallings >