incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Smith <...@iriscouch.com>
Subject Re: Interpreting status of replication from BigCouch
Date Wed, 30 Nov 2011 02:21:23 GMT
FWIW, this is a Javascript term_to_binary implementation. It would be
straightforward to implement binary_to_term. (Patches welcome!)

https://github.com/iriscouch/erlang.js

The key table is this:
https://github.com/iriscouch/erlang.js/blob/master/lib.js#L1-26

On Wed, Nov 30, 2011 at 9:06 AM, Adam Kocoloski <kocolosk@apache.org> wrote:
> Hi Jens, you're certainly right about the format being subject to change -- in BigCouch's
master branch it's [3264, "g1AAA..."] to allow for sane sorting of the sequences.  Just an
FYI.
>
> The hex portion of the sequence is a Base64-encoded term_to_binary representation of
a covering set of shards.  If you execute a binary_to_term(couch_util:decodeBase64Url("g1AAA..."))
on the two sequences in that status string you'll see that the server replaced one of the
shards in the denominator with a replica.  Perhaps one of the nodes in the cluster was unavailable
when the replication request was made.  I'm afraid there's no good way to do an equality
comparison on the client when that happens.  Under normal operating conditions the two hex
blobs should compare equal at the end of a replication.  Regards,
>
> Adam
>
> On Nov 29, 2011, at 5:07 PM, Jens Alfke wrote:
>
>> CouchCocoa attempts to provide progress information about replications, so the app
can display a progress bar or similar UI, or at least indicate when replication has completed.
This is surprisingly difficult. I got it working well with regular CouchDB, but now I’m
re-opening the can of worms because BigCouch doesn’t use integral sequence numbers and so
the status messages provided by the replicator contain binary goop, for example:
>>
>>        “status”: "Processed <<\"3264-g1AAAADzeJzLYWBgYMlgTmFQSElKzi9KdUhJMtLLTS3KLElMT9VLzskvTUnMK9HLSy3JAapkSmRIsv___39WEgMDsyGqNhM82pIcgGRSPUynDvEW5rEASYYGIAXUvB-s24F4eyG6D0B0Q-xWzgIA8ZZODg\">>
/ <<\"3264-g1AAAADzeJzLYWBgYMlgTmFQSElKzi9KdUhJMtLLTS3KLElMT9VLzskvTUnMK9HLSy3JAapkSmRIsv___39WEgMDsyGqNkM82pIcgGRSPUynDvEW5rEASYYGIAXUvB-s2wFVtwlB3QcguiF2K2cBAO-bTgs\">>
changes”
>>
>> I’ve updated my parser to be able to tweeze out the two binary blobs, and I figured
I could at least compare them for string equality to detect when the last change is processed.
Unfortunately this doesn’t appear to work. The example above is the last status message
I got in a pull from Cloudant; apparently it’s completed, because nothing else has been
copied over in the 15 minutes since, but the two blobs are not exactly equal. (They are _almost_
equal, up to the last 25 or so characters.)
>>
>> The unfortunate result is that in the iOS GrocerySync app, the replication progress
bar gets stuck and never goes away.
>>
>> I realize that these blobs are probably deliberately opaque and their format Subject
To Change Without Notice, but it would be helpful if I had some kind of heuristic about their
current format that would at least let me detect when a replication has ended. Any suggestions?
>>
>> —Jens
>>
>
>



-- 
Iris Couch

Mime
View raw message