Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (athena.apache.org: domain of dch@jsonified.com designates
 74.125.83.50 as permitted sender)
Date: Tue, 4 Feb 2014 19:28:27 +0100
From: Dave Cottlehuber <dch@jsonified.com>
To: user@couchdb.apache.org
Message-ID: <etPan.52f1314b.327b23c6.eaca@akai.jsonified.com>
In-Reply-To: 
 <CABu8_fX9QMaHpD+jHSgR95cUG=VYhZkbFgzjPsXgovcbkj0Kkg@mail.gmail.com>
References: 
 <CAE-1UZEAh1K056bnnvPj9VvaOjSThrHthLEVoYq7FFu0aq4FeA@mail.gmail.com>
 <CABu8_fX9QMaHpD+jHSgR95cUG=VYhZkbFgzjPsXgovcbkj0Kkg@mail.gmail.com>
Subject: Re: Replication without IP connection
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

On 29. J=C3=A4nner 2014 at 16:21:37, Brad Rhoads (bdrhoa=40gmail.com) wro=
te:
> =46rom: Brad Rhoads To: user=40couchdb.apache.org Subject: Replication =
=20
> without IP connection Date: 29. J=C3=A4nner 2014 16:21:04 MEZ Our Estan=
te app (http://www.maf.org/estante) =20
> is has similar requirements. Except right now we only do peer to peer. =
Can you schedule =20
> a time for remote users to get together every so often=3F Or can you ge=
t a dedicated person =20
> to visit the remote sights=3F You might check out Open Data Kit. It won=
't help with what you =20
> already built, but it might work over hf. Another solution could be to =
use voice to call =20
> in results, if you have a signal. Maybe the easiest thing would be to h=
ave your app generate =20
> emails to the central location. We developed email over HR years ago. =46=
eel free to contact =20
> me off list. On Jan 27, 2014 9:30 AM, =22Aaron Huslage=22 wrote: > Hi, =
=20
> > > I have a strange problem that I'm hoping someone either has dealt w=
ith or > has ideas =20
> about solving. > > We have developed a medical records system as a Couc=
hApp that allows =20
> > clinicians in remote areas to submit medical records to a local serve=
r. > These records =20
> need to be replicated back to a central office for reporting > purposes=
. > > These remote =20
> clinics are completely disconnected from the Internet and > connect to =
one another over =20
> H=46 Radio (think ham radio back in the day). The > radio links are not=
 really reliable or =20
> quick enough for PPP to work well, > so we are using UUCP instead of IP=
. > > In this scenario, =20
> there is no interactive communication between nodes. > Everything is ba=
tch file copies. =20
> I need a way to do a CouchDB replication > using only file transfers an=
d cron/inotify jobs. =20
> > > Any ideas or scripts would be helpful. > > Thanks, > Aaron > > -- >=
 Aaron Huslage > +1-919-600-1712 =20
> > IM: GTalk - huslage=40gmail.com; Skype - huslage > > PGP: 0x360DE590 =
/ 30D4 B15B 74B2 =460E9 =20
> 35D4 ABE0 2CB7 AC60 360D E590 >

Three things came to mind which are worth mentioning;

1. HTTP and JSON are almost certainly not the best protocol & wire format=
, considering your bandwidth (and presumably =C2=A0power) constraints.

Consider using msgpack=5B1=5D, google protocol buffers =5B2=5D, capnproto=
 =5B3=5D, piqi=5B4=5D anything with a smaller on-the-wire format, that id=
eally takes account of the schema you have implicitly in your data. =5B5=5D=
 might be interesting too, wrt MQTT-S as an alternative lightweight trans=
port too. I=E2=80=99d consider carefully the robustness of the transport =
layer to bit flipping & other errors.

2. I don=E2=80=99t see replication as inherently requiring a synchronous =
question/answer session where both parties need to be online, it=E2=80=99=
s this way in CouchDB because it is a bit more efficient. Consider rollin=
g your own implementation that speaks JSON locally but creates a pack fil=
e using your optimised on-the-wire format. Some implementations I=E2=80=99=
m aware of at =5B6=5D, while these are all old, they are also simple. The=
re=E2=80=99s no doubt a great one tucked inside pouchdb, and presumably i=
n the couchbase sync libraries too for reference. =46inding those is an e=
xcercise to the reader.

If you need to write a replication & wire format for your app from scratc=
h, you might consider reading about CRDTs (commutative replicated datatyp=
es) which offer a robust theoretic basis for merging and ordering data as=
ynchronously. I believe the next version of Riak has support for these.

3. This sure sounds like a solved sort of problem in the HAM radio world.=
 You might find an existing solution with a little pollination. But I=E2=80=
=99m not sure that CouchDB and RESTish/HTTP behaviour are really the best=
 fit for your app wrt to how you need replication to work. But it might b=
e better than anything else. If you do decide to hack in and around couch=
 to make all this work, I=E2=80=99d love to see a plugin that uses a non-=
HTTP transport turn up in the community.

=5B1=5D:=C2=A0https://github.com/msgpack/msgpack-erlang
=5B2=5D:=C2=A0https://github.com/jeremyong/eprotoc &=C2=A0https://github.=
com/basho/erlang=5Fprotobuffs
=5B3=5D:=C2=A0http://kentonv.github.io/capnproto/ &=C2=A0https://github.c=
om/kaos/ecapnp
=5B4=5D: http://piqi.org/
=5B5=5D: http://www.slideshare.net/nivertech/zvi-mqtts-foreuc2013
=5B6=5D:=C2=A0https://github.com/jo/roy-replicator &=C2=A0https://github.=
com/mikeal/replicate and=C2=A0https://github.com/maxogden/replicate

A+
-- =20
Dave Cottlehuber
Sent from my PDP11