Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
From: Adam Kocoloski <kocolosk@apache.org>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_6E4C8AEC-6407-4E6D-80A7-112AD5410D2F"
Subject: CouchDB and Kubernetes
Message-Id: <4C61DA8D-C714-4E64-89B8-145B8A885F7C@apache.org>
Date: Fri, 29 Apr 2016 22:55:00 -0400
To: dev@couchdb.apache.org
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))

--Apple-Mail=_6E4C8AEC-6407-4E6D-80A7-112AD5410D2F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hi all,

I=E2=80=99ve doing a bit of poking around the container orchestration =
space lately and looking at how we might best deploy a CouchDB 2.0 =
cluster in a container environment. In general I=E2=80=99ve been pretty =
impressed with the design point of the Kubernetes project, and I wanted =
to see how hard it would be to put together a proof of concept.

As a preamble, I needed to put together a container image for 2.0 that =
just runs a single Erlang VM instead of the container-local =E2=80=9Cdev =
cluster=E2=80=9D. You can find that work here:

https://github.com/klaemo/docker-couchdb/pull/52 =
<https://github.com/klaemo/docker-couchdb/pull/52>

So far, so good - now for Kubernetes itself. My goal was to figure out =
how to deploy a collection of =E2=80=9CPods=E2=80=9D that could discover =
one another and self-assemble into a cluster. Kubernetes differs from =
the traditional Docker network model in that every Pod gets an IP =
address that is routable from all other Pods in the cluster. As a result =
there=E2=80=99s no need for some of the port gymnastics that one might =
encounter with other Docker environments - each CouchDB pod can listen =
on 5984, 4369 and whatever distribution port you like on its own IP.

What you don=E2=80=99t get with Pods is a hostname that=E2=80=99s =
discoverable from other Pods in the cluster. A =E2=80=9CService=E2=80=9D =
(a replicated, load-balanced collection of Pods) can optionally have a =
DNS name, but the Pods themselves do not. This throws a wrench in the =
most common distributed Erlang setup, where each node gets a name like =
=E2=80=9Ccouchdb@FQDN=E2=80=9D and the FQDNs are resolvable to IP =
addresses via DNS.

It is certainly possible to specify an Erlang node name like =
=E2=80=9Ccouchdb@12.34.56.78 <mailto:couchdb@12.34.56.78>=E2=80=9D, but =
we need to be a bit careful here. CouchDB is currently forcing the =
Erlang node name to do =E2=80=9Cdouble-duty=E2=80=9D; it=E2=80=99s both =
the way that the nodes in a cluster figure out how to route traffic to =
one another and it=E2=80=99s the identifier for nodes to claim ownership =
over individual replicas of database shards in the shard map. Speaking =
from experience it=E2=80=99s often quite useful operationally to remap a =
given Erlang node name to a new server and have the new server be =
automatically populated with the replicas it=E2=80=99s supposed to own. =
If we use the Pod IP in Kubernetes for the node name we won=E2=80=99t =
have that luxury.

I think the best path forward here would be to extend the =E2=80=9CNode" =
concept in a CouchDB cluster so that it has an identifier which is =
allowed to be distinct fro the Erlang node name. The =E2=80=9CCouchDB =
Node=E2=80=9D is the one that owns database shard replicas, and it can =
be remapped to different distributed Erlang nodes over time via =
modification of an attribute in the _nodes DB.

Hope you all found this useful =E2=80=94 I=E2=80=99m quite interested in =
finding way to make it easier for users to acquire a highly-available =
cluster configured in the =E2=80=9Cright way=E2=80=9D, and I think =
projects like Kubernetes have a lot of promise in this regard. Cheers,

Adam=

--Apple-Mail=_6E4C8AEC-6407-4E6D-80A7-112AD5410D2F--