hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marton, Elek" <...@anzix.net>
Subject [DISCUSS] official docker image(s) for hadoop
Date Fri, 08 Sep 2017 12:45:40 GMT

TL;DR: I propose to create official hadoop images and upload them to the 
dockerhub.

GOAL/SCOPE: I would like improve the existing documentation with 
easy-to-use docker based recipes to start hadoop clusters with various 
configuration.

The images also could be used to test experimental features. For example 
ozone could be tested easily with these compose file and configuration:

https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6

Or even the configuration could be included in the compose file:

https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml

I would like to create separated example compose files for federation, 
ha, metrics usage, etc. to make it easier to try out and understand the 
features.

CONTEXT: There is an existing Jira 
https://issues.apache.org/jira/browse/HADOOP-13397
But it’s about a tool to generate production quality docker images 
(multiple types, in a flexible way). If no objections, I will create a 
separated issue to create simplified docker images for rapid prototyping 
and investigating new features. And register the branch to the dockerhub 
to create the images automatically.

MY BACKGROUND: I am working with docker based hadoop/spark clusters 
quite a while and run them succesfully in different environments 
(kubernetes, docker-swarm, nomad-based scheduling, etc.) My work is 
available from here: https://github.com/flokkr but they could handle 
more complex use cases (eg. instrumenting java processes with btrace, or 
read/reload configuration from consul).
  And IMHO in the official hadoop documentation it’s better to suggest 
to use official apache docker images and not external ones (which could 
be changed).

Please let me know if you have any comments.

Marton

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message