hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wangda Tan <wheele...@gmail.com>
Subject Re: [DISCUSS] official docker image(s) for hadoop
Date Wed, 13 Sep 2017 18:19:06 GMT
+1 to add Hadoop docker image for easier testing / prototyping, it gonna be
super helpful!

Thanks,
Wangda

On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
miklos.szegedi@cloudera.com> wrote:

> Marton, thank you for working on this. I think Official Docker images for
> Hadoop would be very useful for a lot of reasons. I think that it is better
> to have a coordinated effort with production ready base images with
> dependent images for prototyping. Does anyone else have an opinion about
> this?
>
> Thank you,
> Miklos
>
> On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek <hdp@anzix.net> wrote:
>
> >
> > TL;DR: I propose to create official hadoop images and upload them to the
> > dockerhub.
> >
> > GOAL/SCOPE: I would like improve the existing documentation with
> > easy-to-use docker based recipes to start hadoop clusters with various
> > configuration.
> >
> > The images also could be used to test experimental features. For example
> > ozone could be tested easily with these compose file and configuration:
> >
> > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> >
> > Or even the configuration could be included in the compose file:
> >
> > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > ker-compose.yaml
> >
> > I would like to create separated example compose files for federation,
> ha,
> > metrics usage, etc. to make it easier to try out and understand the
> > features.
> >
> > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > /browse/HADOOP-13397
> > But it’s about a tool to generate production quality docker images
> > (multiple types, in a flexible way). If no objections, I will create a
> > separated issue to create simplified docker images for rapid prototyping
> > and investigating new features. And register the branch to the dockerhub
> to
> > create the images automatically.
> >
> > MY BACKGROUND: I am working with docker based hadoop/spark clusters quite
> > a while and run them succesfully in different environments (kubernetes,
> > docker-swarm, nomad-based scheduling, etc.) My work is available from
> here:
> > https://github.com/flokkr but they could handle more complex use cases
> > (eg. instrumenting java processes with btrace, or read/reload
> configuration
> > from consul).
> >  And IMHO in the official hadoop documentation it’s better to suggest to
> > use official apache docker images and not external ones (which could be
> > changed).
> >
> > Please let me know if you have any comments.
> >
> > Marton
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message