hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhe Zhang <zhe.zhang.resea...@gmail.com>
Subject Re: Multiple node Hadoop cluster on Docker for test and debugging
Date Wed, 09 Nov 2016 17:30:35 GMT
Nice work Kai! Quick comment: have you considered Bigtop? It allows
creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
a custom built Hadoop (from the instructions I read, it builds from a
certain git branch; I'm not sure if that customizable).

On Wed, Nov 9, 2016 at 5:39 AM Sasaki Kai <lewuathe@me.com> wrote:

> Hi Hadoop developers
>
> The other day I created a tool for launching multiple node hadoop cluster
> on docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source
> code.
> It is useful for testing and debugging. Actually I often use it before
> submitting a patch to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster <
> https://github.com/Lewuathe/docker-hadoop-cluster>
>
> And I also updated to build the latest trunk image automatically and
> upload onto Docker Hub.
> So you can easily check and test the latest trunk branch in the
> environment which is more close to actual usage.
>
> If you already installed docker and docker-compose, what needed is
> docker-compose.yml like this.
>
> version: '2'
>
> services:
>   master:
>     image: lewuathe/hadoop-master
>     ports:
>       - "9870:9870"
>       - "8088:8088"
>       - "19888:19888"
>       - "8188:8188"
>     container_name: "master"
>   slave1:
>     image: lewuathe/hadoop-slave
>     container_name: "slave1"
>     depends_on:
>       - master
>     ports:
>       - "9901:9864"
>       - "8041:8042"
>   slave2:
>     image: lewuathe/hadoop-slave
>     container_name: "slave2"
>     depends_on:
>       - master
>     ports:
>       - "9902:9864"
>       - "8042:8042"
>
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>
>
> I would be glad if you use this tool for developing and debugging and make
> our development more efficient.
> Please give me any feedbacks to me. Thanks you!
>
> ————————
> Kai Sasaki
> mail: lewuathe@me.com <mailto:lewuathe@me.com>
> github: https://github.com/Lewuathe <https://github.com/Lewuathe>
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message