From commits-return-1071-archive-asf-public=cust-asf.ponee.io@superset.incubator.apache.org Sun Jun 10 06:26:50 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BFA0D180672 for ; Sun, 10 Jun 2018 06:26:49 +0200 (CEST) Received: (qmail 97691 invoked by uid 500); 10 Jun 2018 04:26:48 -0000 Mailing-List: contact commits-help@superset.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@superset.incubator.apache.org Delivered-To: mailing list commits@superset.incubator.apache.org Received: (qmail 97682 invoked by uid 99); 10 Jun 2018 04:26:48 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jun 2018 04:26:48 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id C05DE8281F; Sun, 10 Jun 2018 04:26:47 +0000 (UTC) Date: Sun, 10 Jun 2018 04:26:47 +0000 To: "commits@superset.apache.org" Subject: [incubator-superset] branch master updated: Init docker for local development environment. (#4193) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <152860480753.1504.5263864575924886484@gitbox.apache.org> From: maximebeauchemin@apache.org X-Git-Host: gitbox.apache.org X-Git-Repo: incubator-superset X-Git-Refname: refs/heads/master X-Git-Reftype: branch X-Git-Oldrev: 7d1c035658919473c251bef857f59d8df96fe3d4 X-Git-Newrev: 0a276ff75dcba82efa74f38dff2a58c994fd104f X-Git-Rev: 0a276ff75dcba82efa74f38dff2a58c994fd104f X-Git-NotificationType: ref_changed_plus_diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated This is an automated email from the ASF dual-hosted git repository. maximebeauchemin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-superset.git The following commit(s) were added to refs/heads/master by this push: new 0a276ff Init docker for local development environment. (#4193) 0a276ff is described below commit 0a276ff75dcba82efa74f38dff2a58c994fd104f Author: Xiao Hanyu AuthorDate: Sun Jun 10 12:26:41 2018 +0800 Init docker for local development environment. (#4193) This commit will try to dockerize superset in local development environment. The basic design is: - Enable superset, redis and postgres service instead of using sqlite, just want to simulate production environment settings - Use environment variables to config various app settings. It's easy to run and config superset to any environment if we use environment than traditional config files - For local development environment, we just expose postgres and redis to local host machine thus you can connect local port via `psql` or `redis-cli` - Wrap start up command in a standard `docker-entrypoint.sh`, and use `tail -f /dev/null` combined with manually `superset runserver -d` to make sure that code error didn't cause the container to fail. - Use volumes to share code between host and container, thus you can use your favourite tools to modify code and your code will run in containerized environment - Use volumes to persistent postgres and redis data, and also `node_modules` data. - If we don't cache `node_modules` in docker volume, then every time run docker build, the `node_modules` directory, will is about 500 MB large, will be sent to docker daemon, and make the build quite slow. - Wrap initialization commands to a single script `docker-init.sh` After this dockerize setup, any developers who want to contribute to superset, just follow three easy steps: ``` git clone https://github.com/apache/incubator-superset/ cd incubator-superset cp contrib/docker/{docker-build.sh,docker-compose.yml,docker-entrypoint.sh,docker-init.sh,Dockerfile} . cp contrib/docker/superset_config.py superset/ bash -x docker-build.sh docker-compose up -d docker-compose exec superset bash bash docker-init.sh ``` --- .gitignore | 7 +++++ contrib/docker/Dockerfile | 60 +++++++++++++++++++++++++++++++++++++ contrib/docker/docker-build.sh | 5 ++++ contrib/docker/docker-compose.yml | 48 +++++++++++++++++++++++++++++ contrib/docker/docker-entrypoint.sh | 12 ++++++++ contrib/docker/docker-init.sh | 24 +++++++++++++++ contrib/docker/superset_config.py | 48 +++++++++++++++++++++++++++++ docs/installation.rst | 21 +++++++++++++ 8 files changed, 225 insertions(+) diff --git a/.gitignore b/.gitignore index df3cb8b..11929a9 100644 --- a/.gitignore +++ b/.gitignore @@ -41,3 +41,10 @@ superset/assets/version_info.json *.iml venv @eaDir/ + +# docker +/Dockerfile +/docker-build.sh +/docker-compose.yml +/docker-entrypoint.sh +/docker-init.sh diff --git a/contrib/docker/Dockerfile b/contrib/docker/Dockerfile new file mode 100644 index 0000000..3d47486 --- /dev/null +++ b/contrib/docker/Dockerfile @@ -0,0 +1,60 @@ +FROM python:3.6 + +MAINTAINER Xiao Hanyu + +# Add a normal user +RUN useradd --user-group --create-home --shell /bin/bash work + +# Configure environment +ENV LANG=C.UTF-8 \ + LC_ALL=C.UTF-8 \ + HOME=/home/work + +RUN apt-get update -y + +# Install some dependencies +# http://airbnb.io/superset/installation.html#os-dependencies +RUN apt-get update -y && apt-get install -y build-essential libssl-dev \ + libffi-dev python3-dev libsasl2-dev libldap2-dev + +RUN apt-get install -y vim less postgresql-client redis-tools + +# Install nodejs for custom build +# https://github.com/apache/incubator-superset/blob/master/docs/installation.rst#making-your-own-build +# https://nodejs.org/en/download/package-manager/ +RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - +RUN apt-get install -y nodejs +RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -; \ + echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list; \ + apt-get update; \ + apt-get install -y yarn + +RUN mkdir $HOME/incubator-superset + +WORKDIR $HOME/incubator-superset + +COPY ./ ./ + +RUN pip install --upgrade setuptools pip +RUN pip install -e . && pip install -r requirements-dev.txt + +ENV PATH=/home/work/incubator-superset/superset/bin:$PATH \ + PYTHONPATH=./superset/:$PYTHONPATH + +COPY docker-entrypoint.sh /usr/local/bin/ +RUN chmod +x /usr/local/bin/docker-entrypoint.sh +RUN ln -s usr/local/bin/docker-entrypoint.sh /entrypoint.sh # backwards compat + +COPY ./superset ./superset +RUN chown -R work:work $HOME + +USER work + +RUN cd superset/assets && yarn +RUN cd superset/assets && npm run build + +HEALTHCHECK CMD ["curl", "-f", "http://localhost:8088/health"] + +ENTRYPOINT ["docker-entrypoint.sh"] + +EXPOSE 8088 diff --git a/contrib/docker/docker-build.sh b/contrib/docker/docker-build.sh new file mode 100644 index 0000000..55f7327 --- /dev/null +++ b/contrib/docker/docker-build.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash + +set -ex + +docker build -t apache/incubator-superset -f Dockerfile . diff --git a/contrib/docker/docker-compose.yml b/contrib/docker/docker-compose.yml new file mode 100644 index 0000000..9085793 --- /dev/null +++ b/contrib/docker/docker-compose.yml @@ -0,0 +1,48 @@ +version: '3' +services: + redis: + image: redis:3.2 + restart: always + ports: + - 6379:6379 + volumes: + - redis:/data + postgres: + image: postgres:10 + restart: always + environment: + POSTGRES_DB: superset + POSTGRES_PASSWORD: superset + POSTGRES_USER: superset + ports: + - 5432:5432 + volumes: + - postgres:/var/lib/postgresql/data + superset: + image: apache/incubator-superset + restart: always + environment: + POSTGRES_DB: superset + POSTGRES_USER: superset + POSTGRES_PASSWORD: superset + POSTGRES_HOST: postgres + POSTGRES_PORT: 5432 + REDIS_HOST: redis + REDIS_PORT: 6379 + SUPERSET_ENV: local + ports: + - 8088:8088 + command: "tail -f /dev/null" + depends_on: + - postgres + - redis + volumes: + - .:/home/work/incubator-superset + - superset-node-modules:/home/work/incubator-superset/superset/assets/node_modules +volumes: + postgres: + external: false + redis: + external: false + superset-node-modules: + external: false diff --git a/contrib/docker/docker-entrypoint.sh b/contrib/docker/docker-entrypoint.sh new file mode 100644 index 0000000..c6629c7 --- /dev/null +++ b/contrib/docker/docker-entrypoint.sh @@ -0,0 +1,12 @@ +#!/bin/bash +set -ex + +if [ "$#" -ne 0 ]; then + exec "$@" +elif [ "$SUPERSET_ENV" = "local" ]; then + superset runserver -d +elif [ "$SUPERSET_ENV" = "production" ]; then + superset runserver -a 0.0.0.0 -w $((2 * $(getconf _NPROCESSORS_ONLN) + 1)) +else + superset --help +fi diff --git a/contrib/docker/docker-init.sh b/contrib/docker/docker-init.sh new file mode 100644 index 0000000..940ad4f --- /dev/null +++ b/contrib/docker/docker-init.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash + +set -ex + +# Create an admin user (you will be prompted to set username, first and last name before setting a password) +fabmanager create-admin --app superset + +# Initialize the database +superset db upgrade + +# Load some data to play with +superset load_examples + +# Create default roles and permissions +superset init + +# Need to run `npm run build` when enter contains for first time +cd superset/assets && npm run build && cd ../../ + +# Start superset worker for SQL Lab +superset worker & + +# To start a development web server, use the -d switch +superset runserver -d diff --git a/contrib/docker/superset_config.py b/contrib/docker/superset_config.py new file mode 100644 index 0000000..52536d6 --- /dev/null +++ b/contrib/docker/superset_config.py @@ -0,0 +1,48 @@ +# -*- coding: utf-8 -*- +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals + +import os + + +def get_env_variable(var_name, default=None): + """Get the environment variable or raise exception.""" + try: + return os.environ[var_name] + except KeyError: + if default is not None: + return default + else: + error_msg = 'The environment variable {} was missing, abort...'\ + .format(var_name) + raise EnvironmentError(error_msg) + + +POSTGRES_USER = get_env_variable('POSTGRES_USER') +POSTGRES_PASSWORD = get_env_variable('POSTGRES_PASSWORD') +POSTGRES_HOST = get_env_variable('POSTGRES_HOST') +POSTGRES_PORT = get_env_variable('POSTGRES_PORT') +POSTGRES_DB = get_env_variable('POSTGRES_DB') + +# The SQLAlchemy connection string. +SQLALCHEMY_DATABASE_URI = 'postgresql://%s:%s@%s:%s/%s' % (POSTGRES_USER, + POSTGRES_PASSWORD, + POSTGRES_HOST, + POSTGRES_PORT, + POSTGRES_DB) + +REDIS_HOST = get_env_variable('REDIS_HOST') +REDIS_PORT = get_env_variable('REDIS_PORT') + + +class CeleryConfig(object): + BROKER_URL = 'redis://%s:%s/0' % (REDIS_HOST, REDIS_PORT) + CELERY_IMPORTS = ('superset.sql_lab', ) + CELERY_RESULT_BACKEND = 'redis://%s:%s/1' % (REDIS_HOST, REDIS_PORT) + CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}} + CELERY_TASK_PROTOCOL = 1 + + +CELERY_CONFIG = CeleryConfig diff --git a/docs/installation.rst b/docs/installation.rst index 28efb46..4666f82 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -35,6 +35,27 @@ The Superset web server and the Superset Celery workers (optional) are stateless, so you can scale out by running on as many servers as needed. +Start with Docker +----------------- + +If you know docker, then you're lucky, we have shortcut road for you to +initialize development environment: :: + + git clone https://github.com/apache/incubator-superset/ + cd incubator-superset + cp contrib/docker/{docker-build.sh,docker-compose.yml,docker-entrypoint.sh,docker-init.sh,Dockerfile} . + cp contrib/docker/superset_config.py superset/ + bash -x docker-build.sh + docker-compose up -d + docker-compose exec superset bash + bash docker-init.sh + +After several minutes for sueprset initialization to finish, you can open a +a browser and view `http://localhost:8088` to start your journey. + +Or if you're curious and want to install superset from bottom up, then go +ahead. + OS dependencies --------------- -- To stop receiving notification emails like this one, please contact maximebeauchemin@apache.org.