hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "dineshs/DockerNetworkingForYarnApps" by dineshs
Date Thu, 12 Jun 2014 22:20:11 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "dineshs/DockerNetworkingForYarnApps" page has been changed by dineshs:
https://wiki.apache.org/hadoop/dineshs/DockerNetworkingForYarnApps?action=diff&rev1=1&rev2=2

Comment:
Add host network approach, available in Docker 1.0

  
  In the way of some quick background (for the benefit of Docker guys who may be unfamiliar),
YARN applications consist of a set of Docker containers with one of them running the Application
Master (AM) and the rest running its tasks.  YARN resource manager (RM) launches AM which
acts as the focal point for the application.  Typically, AM starts listening on a dynamic
port, launches the task containers and passes them application configuration.  In particular,
the configuration includes the IP address and port where AM listens.  The AM container and
its task containers could be scheduled on different hosts in the cluster based on data locality,
resource availability etc.
  
- In a multitenant cluster, applications belonging to different tenants should be securely
isolated such that tenants would not be able to snoop each others traffic.
+ In a multitenant cluster, applications belonging to different tenants should be securely
isolated to prevent them from snooping each others traffic.
  
  == Possible approaches ==
  
@@ -19, +19 @@

  
  Default Docker networking based on NAT'ed interfaces doesn't work well for inter-host container
networking.  One possibility, based on mechanisms currently supported by Docker, is to expose
inbound container ports to the host and have application components talk to one another through
their respective hosts.  The problem though is that the port on which the application master
listens on is not known when its container is created.  Since Docker only supports exposing
ports at the time of container creation, this option won't work.  Conceivably, exposing ports
on the fly can be implemented.  It requires an API between the application and YARN/Docker
to communicate the ports to be exposed.  Even that won't help with existing YARN applications
in the wild that expect seamless connectivity among their components.
  
- === Connect application containers across hosts into an L2 subnet ===
+ === Open vSwitch based L2 subnet ===
  
  [[https://raw.githubusercontent.com/dineshs-altiscale/yarn-apps-in-docker/master/pictures/net-1.png|This
figure]] shows the network topology.  The IP address space of [[https://github.com/dotcloud/docker/pull/6101|Docker
subnet is partitioned]] among the hosts and containers on a particular host are assigned IP
address from their partition.  The host level bridges can then be weaved together into an
L2-over-L3 Open vSwitch subnet through point-to-point GRE tunnels.
  
@@ -31, +31 @@

  
  If Docker natively supports OVS bridging, that would avoid an additional hop between the
Linux bridges and the OVS bridge.
  
+ === Use host networking ===
+ 
+ Docker 1.0 supports disabling network namespace for containers such that they simply share
the host network including its IP address and port space.  It provides a simple working solution
if network isolation is not a concern.
+ 
+ === 
+ 

Mime
View raw message