hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasaki Kai <lewua...@me.com>
Subject Multiple node Hadoop cluster on Docker for test and debugging
Date Wed, 09 Nov 2016 13:38:55 GMT
Hi Hadoop developers

The other day I created a tool for launching multiple node hadoop cluster on docker container.
You can easily launch multiple node hadoop cluster from your Hadoop source code. 
It is useful for testing and debugging. Actually I often use it before submitting a patch
to Hadoop project.
https://github.com/Lewuathe/docker-hadoop-cluster <https://github.com/Lewuathe/docker-hadoop-cluster>

And I also updated to build the latest trunk image automatically and upload onto Docker Hub.
So you can easily check and test the latest trunk branch in the environment which is more
close to actual usage.

If you already installed docker and docker-compose, what needed is docker-compose.yml like
this.

version: '2'

services:
  master:
    image: lewuathe/hadoop-master
    ports:
      - "9870:9870"
      - "8088:8088"
      - "19888:19888"
      - "8188:8188"
    container_name: "master"
  slave1:
    image: lewuathe/hadoop-slave
    container_name: "slave1"
    depends_on:
      - master
    ports:
      - "9901:9864"
      - "8041:8042"
  slave2:
    image: lewuathe/hadoop-slave
    container_name: "slave2"
    depends_on:
      - master
    ports:
      - "9902:9864"
      - "8042:8042"

The usage in detail is described in the repository.
https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>

I would be glad if you use this tool for developing and debugging and make our development
more efficient.
Please give me any feedbacks to me. Thanks you!

————————
Kai Sasaki
mail: lewuathe@me.com <mailto:lewuathe@me.com>
github: https://github.com/Lewuathe <https://github.com/Lewuathe>



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message