hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingliang Liu <lium...@apache.org>
Subject Re: Multiple node Hadoop cluster on Docker for test and debugging
Date Wed, 09 Nov 2016 19:15:51 GMT
I know a similar tool; it has Ambari and Spark integration as well.

https://github.com/weiqingy/caochong <https://github.com/weiqingy/caochong>

Thanks,

L

> On Nov 9, 2016, at 5:38 AM, Sasaki Kai <lewuathe@me.com> wrote:
> 
> Hi Hadoop developers
> 
> The other day I created a tool for launching multiple node hadoop cluster on docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source code. 
> It is useful for testing and debugging. Actually I often use it before submitting a patch
to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster <https://github.com/Lewuathe/docker-hadoop-cluster>
> 
> And I also updated to build the latest trunk image automatically and upload onto Docker
Hub.
> So you can easily check and test the latest trunk branch in the environment which is
more close to actual usage.
> 
> If you already installed docker and docker-compose, what needed is docker-compose.yml
like this.
> 
> version: '2'
> 
> services:
>  master:
>    image: lewuathe/hadoop-master
>    ports:
>      - "9870:9870"
>      - "8088:8088"
>      - "19888:19888"
>      - "8188:8188"
>    container_name: "master"
>  slave1:
>    image: lewuathe/hadoop-slave
>    container_name: "slave1"
>    depends_on:
>      - master
>    ports:
>      - "9901:9864"
>      - "8041:8042"
>  slave2:
>    image: lewuathe/hadoop-slave
>    container_name: "slave2"
>    depends_on:
>      - master
>    ports:
>      - "9902:9864"
>      - "8042:8042"
> 
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>
> 
> I would be glad if you use this tool for developing and debugging and make our development
more efficient.
> Please give me any feedbacks to me. Thanks you!
> 
> ————————
> Kai Sasaki
> mail: lewuathe@me.com <mailto:lewuathe@me.com>
> github: https://github.com/Lewuathe <https://github.com/Lewuathe>
> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message