couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kocoloski <kocol...@apache.org>
Subject Re: Interpreting status of replication from BigCouch
Date Wed, 30 Nov 2011 02:37:57 GMT
Nifty!  I should also mention that the BigCouch sequence is gzip'ed, which the Erlang binary_to_term
implementation handles transparently.  Regards,

Adam

On Nov 29, 2011, at 9:21 PM, Jason Smith wrote:

> FWIW, this is a Javascript term_to_binary implementation. It would be
> straightforward to implement binary_to_term. (Patches welcome!)
> 
> https://github.com/iriscouch/erlang.js
> 
> The key table is this:
> https://github.com/iriscouch/erlang.js/blob/master/lib.js#L1-26
> 
> On Wed, Nov 30, 2011 at 9:06 AM, Adam Kocoloski <kocolosk@apache.org> wrote:
>> Hi Jens, you're certainly right about the format being subject to change -- in BigCouch's
master branch it's [3264, "g1AAA..."] to allow for sane sorting of the sequences.  Just an
FYI.
>> 
>> The hex portion of the sequence is a Base64-encoded term_to_binary representation
of a covering set of shards.  If you execute a binary_to_term(couch_util:decodeBase64Url("g1AAA..."))
on the two sequences in that status string you'll see that the server replaced one of the
shards in the denominator with a replica.  Perhaps one of the nodes in the cluster was unavailable
when the replication request was made.  I'm afraid there's no good way to do an equality comparison
on the client when that happens.  Under normal operating conditions the two hex blobs should
compare equal at the end of a replication.  Regards,
>> 
>> Adam
>> 
>> On Nov 29, 2011, at 5:07 PM, Jens Alfke wrote:
>> 
>>> CouchCocoa attempts to provide progress information about replications, so the
app can display a progress bar or similar UI, or at least indicate when replication has completed.
This is surprisingly difficult. I got it working well with regular CouchDB, but now I’m
re-opening the can of worms because BigCouch doesn’t use integral sequence numbers and so
the status messages provided by the replicator contain binary goop, for example:
>>> 
>>>        “status”: "Processed <<\"3264-g1AAAADzeJzLYWBgYMlgTmFQSElKzi9KdUhJMtLLTS3KLElMT9VLzskvTUnMK9HLSy3JAapkSmRIsv___39WEgMDsyGqNhM82pIcgGRSPUynDvEW5rEASYYGIAXUvB-s24F4eyG6D0B0Q-xWzgIA8ZZODg\">>
/ <<\"3264-g1AAAADzeJzLYWBgYMlgTmFQSElKzi9KdUhJMtLLTS3KLElMT9VLzskvTUnMK9HLSy3JAapkSmRIsv___39WEgMDsyGqNkM82pIcgGRSPUynDvEW5rEASYYGIAXUvB-s2wFVtwlB3QcguiF2K2cBAO-bTgs\">>
changes”
>>> 
>>> I’ve updated my parser to be able to tweeze out the two binary blobs, and I
figured I could at least compare them for string equality to detect when the last change is
processed. Unfortunately this doesn’t appear to work. The example above is the last status
message I got in a pull from Cloudant; apparently it’s completed, because nothing else has
been copied over in the 15 minutes since, but the two blobs are not exactly equal. (They are
_almost_ equal, up to the last 25 or so characters.)
>>> 
>>> The unfortunate result is that in the iOS GrocerySync app, the replication progress
bar gets stuck and never goes away.
>>> 
>>> I realize that these blobs are probably deliberately opaque and their format
Subject To Change Without Notice, but it would be helpful if I had some kind of heuristic
about their current format that would at least let me detect when a replication has ended.
Any suggestions?
>>> 
>>> —Jens
>>> 
>> 
>> 
> 
> 
> 
> -- 
> Iris Couch


Mime
View raw message