couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Cottlehuber <...@jsonified.com>
Subject Re: Replication without IP connection
Date Tue, 04 Feb 2014 18:28:27 GMT
On 29. Jänner 2014 at 16:21:37, Brad Rhoads (bdrhoa@gmail.com) wrote:
> From: Brad Rhoads To: user@couchdb.apache.org Subject: Replication  
> without IP connection Date: 29. Jänner 2014 16:21:04 MEZ Our Estante app (http://www.maf.org/estante)
 
> is has similar requirements. Except right now we only do peer to peer. Can you schedule
 
> a time for remote users to get together every so often? Or can you get a dedicated person
 
> to visit the remote sights? You might check out Open Data Kit. It won't help with what
you  
> already built, but it might work over hf. Another solution could be to use voice to call
 
> in results, if you have a signal. Maybe the easiest thing would be to have your app generate
 
> emails to the central location. We developed email over HR years ago. Feel free to contact
 
> me off list. On Jan 27, 2014 9:30 AM, "Aaron Huslage" wrote: > Hi,  
> > > I have a strange problem that I'm hoping someone either has dealt with or >
has ideas  
> about solving. > > We have developed a medical records system as a CouchApp that
allows  
> > clinicians in remote areas to submit medical records to a local server. > These
records  
> need to be replicated back to a central office for reporting > purposes. > >
These remote  
> clinics are completely disconnected from the Internet and > connect to one another
over  
> HF Radio (think ham radio back in the day). The > radio links are not really reliable
or  
> quick enough for PPP to work well, > so we are using UUCP instead of IP. > >
In this scenario,  
> there is no interactive communication between nodes. > Everything is batch file copies.
 
> I need a way to do a CouchDB replication > using only file transfers and cron/inotify
jobs.  
> > > Any ideas or scripts would be helpful. > > Thanks, > Aaron > >
-- > Aaron Huslage > +1-919-600-1712  
> > IM: GTalk - huslage@gmail.com; Skype - huslage > > PGP: 0x360DE590 / 30D4
B15B 74B2 F0E9  
> 35D4 ABE0 2CB7 AC60 360D E590 >

Three things came to mind which are worth mentioning;

1. HTTP and JSON are almost certainly not the best protocol & wire format, considering
your bandwidth (and presumably  power) constraints.

Consider using msgpack[1], google protocol buffers [2], capnproto [3], piqi[4] anything with
a smaller on-the-wire format, that ideally takes account of the schema you have implicitly
in your data. [5] might be interesting too, wrt MQTT-S as an alternative lightweight transport
too. I’d consider carefully the robustness of the transport layer to bit flipping &
other errors.

2. I don’t see replication as inherently requiring a synchronous question/answer session
where both parties need to be online, it’s this way in CouchDB because it is a bit more
efficient. Consider rolling your own implementation that speaks JSON locally but creates a
pack file using your optimised on-the-wire format. Some implementations I’m aware of at
[6], while these are all old, they are also simple. There’s no doubt a great one tucked
inside pouchdb, and presumably in the couchbase sync libraries too for reference. Finding
those is an excercise to the reader.

If you need to write a replication & wire format for your app from scratch, you might
consider reading about CRDTs (commutative replicated datatypes) which offer a robust theoretic
basis for merging and ordering data asynchronously. I believe the next version of Riak has
support for these.

3. This sure sounds like a solved sort of problem in the HAM radio world. You might find an
existing solution with a little pollination. But I’m not sure that CouchDB and RESTish/HTTP
behaviour are really the best fit for your app wrt to how you need replication to work. But
it might be better than anything else. If you do decide to hack in and around couch to make
all this work, I’d love to see a plugin that uses a non-HTTP transport turn up in the community.

[1]: https://github.com/msgpack/msgpack-erlang
[2]: https://github.com/jeremyong/eprotoc & https://github.com/basho/erlang_protobuffs
[3]: http://kentonv.github.io/capnproto/ & https://github.com/kaos/ecapnp
[4]: http://piqi.org/
[5]: http://www.slideshare.net/nivertech/zvi-mqtts-foreuc2013
[6]: https://github.com/jo/roy-replicator & https://github.com/mikeal/replicate and https://github.com/maxogden/replicate

A+
--  
Dave Cottlehuber
Sent from my PDP11

Mime
View raw message