predictionio-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shins...@apache.org
Subject [predictionio] branch develop updated: [PIO-86] Docker support (#462)
Date Wed, 10 Oct 2018 07:31:04 GMT
This is an automated email from the ASF dual-hosted git repository.

shinsuke pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/predictionio.git


The following commit(s) were added to refs/heads/develop by this push:
     new dc08628  [PIO-86] Docker support (#462)
dc08628 is described below

commit dc08628eb6489e594af369310bb1c9d36c81d15a
Author: Shinsuke Sugaya <shinsuke.sugaya@gmail.com>
AuthorDate: Wed Oct 10 16:30:57 2018 +0900

    [PIO-86] Docker support (#462)
---
 docker/README.md                              | 249 ++++++++++++++++++++++++++
 docker/bin/pio-docker                         |  56 ++++++
 docker/deploy/run.sh                          |  21 +++
 docker/docker-compose.deploy.yml              |  23 +++
 docker/docker-compose.spark.yml               |  34 ++++
 docker/docker-compose.yml                     |  25 +++
 docker/elasticsearch/docker-compose.base.yml  |  41 +++++
 docker/elasticsearch/docker-compose.event.yml |  22 +++
 docker/elasticsearch/docker-compose.meta.yml  |  22 +++
 docker/localfs/docker-compose.model.yml       |  27 +++
 docker/mysql/docker-compose.base.yml          |  34 ++++
 docker/mysql/docker-compose.event.yml         |  22 +++
 docker/mysql/docker-compose.meta.yml          |  22 +++
 docker/mysql/docker-compose.model.yml         |  22 +++
 docker/pgsql/docker-compose.base.yml          |  32 ++++
 docker/pgsql/docker-compose.event.yml         |  22 +++
 docker/pgsql/docker-compose.meta.yml          |  22 +++
 docker/pgsql/docker-compose.model.yml         |  22 +++
 docker/pio/Dockerfile                         |  69 +++++++
 docker/pio/pio_run                            |  64 +++++++
 docker/templates/.keep                        |  14 ++
 21 files changed, 865 insertions(+)

diff --git a/docker/README.md b/docker/README.md
new file mode 100644
index 0000000..f1bc5a0
--- /dev/null
+++ b/docker/README.md
@@ -0,0 +1,249 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+Apache PredictionIO Docker
+==========================
+
+## Overview
+
+PredictionIO Docker provides Docker image for use in development and production environment.
+
+
+## Usage
+
+### Run PredictionIO with Selectable docker-compose Files
+
+You can choose storages for event/meta/model to select docker-compose.yml.
+
+```
+docker-compose -f docker-compose.yml -f ... up
+```
+
+Supported storages are as below:
+
+| Type  | Storage                          |
+|:-----:|:---------------------------------|
+| Event | Postgresql, MySQL, Elasticsearch |
+| Meta  | Postgresql, MySQL, Elasticsearch |
+| Model | Postgresql, MySQL, LocalFS       |
+
+If you run PredictionIO with Postgresql, run as below:
+
+```
+docker-compose -f docker-compose.yml \
+  -f pgsql/docker-compose.base.yml \
+  -f pgsql/docker-compose.meta.yml \
+  -f pgsql/docker-compose.event.yml \
+  -f pgsql/docker-compose.model.yml \
+  up
+```
+
+To use localfs as model storage, change as below:
+
+```
+docker-compose -f docker-compose.yml \
+  -f pgsql/docker-compose.base.yml \
+  -f pgsql/docker-compose.meta.yml \
+  -f pgsql/docker-compose.event.yml \
+  -f localfs/docker-compose.model.yml \
+  up
+```
+
+## Tutorial
+
+In this demo, we will show you how to build a recommendation template.
+
+### Run PredictionIO environment
+
+The following command starts PredictionIO with an event server.
+PredictionIO docker image mounts ./templates directory to /templates.
+
+```
+$ docker-compose -f docker-compose.yml \
+    -f pgsql/docker-compose.base.yml \
+    -f pgsql/docker-compose.meta.yml \
+    -f pgsql/docker-compose.event.yml \
+    -f pgsql/docker-compose.model.yml \
+    up
+```
+
+We provide `pio-docker` command as an utility for `pio` command.
+`pio-docker` invokes `pio` command in PredictionIO container.
+
+```
+$ export PATH=`pwd`/bin:$PATH
+$ pio-docker status
+...
+[INFO] [Management$] Your system is all ready to go.
+```
+
+### Download Recommendation Template
+
+This demo uses [predictionio-template-recommender](https://github.com/apache/predictionio-template-recommender).
+
+```
+$ cd templates
+$ git clone https://github.com/apache/predictionio-template-recommender.git MyRecommendation
+$ cd MyRecommendation
+```
+
+### Register Application
+
+You need to register this application to PredictionIO:
+
+```
+$ pio-docker app new MyApp1
+[INFO] [App$] Initialized Event Store for this app ID: 1.
+[INFO] [Pio$] Created a new app:
+[INFO] [Pio$]       Name: MyApp1
+[INFO] [Pio$]         ID: 1
+[INFO] [Pio$] Access Key: i-zc4EleEM577EJhx3CzQhZZ0NnjBKKdSbp3MiR5JDb2zdTKKzH9nF6KLqjlMnvl
+```
+
+Since an access key is required in subsequent steps, set it to ACCESS_KEY.
+
+```
+$ ACCESS_KEY=i-zc4EleEM577EJhx3CzQhZZ0NnjBKKdSbp3MiR5JDb2zdTKKzH9nF6KLqjlMnvl
+```
+
+`engine.json` contains an application name, so replace `INVALID_APP_NAME` with `MyApp1`.
+
+```
+...
+"datasource": {
+  "params" : {
+    "appName": "MyApp1"
+  }
+},
+...
+```
+
+### Import Data
+
+To import training data to Event server for PredictionIO, this template provides an import
tool.
+The tool depends on PredictionIO Python SDK and install as below:
+
+```
+$ pip install predictionio
+```
+and then import data:
+```
+$ curl https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_movielens_data.txt
--create-dirs -o data/sample_movielens_data.txt
+$ python data/import_eventserver.py --access_key $ACCESS_KEY
+```
+
+### Build Template
+
+This is Scala based template.
+So, you need to build this template by `pio` command.
+
+```
+$ pio-docker build --verbose
+```
+
+### Train and Create Model
+
+To train a recommendation model, run `train` sub-command:
+
+```
+$ pio-docker train
+```
+
+### Deploy Model
+
+If a recommendation model is created successfully, deploy it to Prediction server for PredictionIO.
+
+```
+$ pio-docker deploy
+
+```
+You can check predictions as below:
+```
+$ curl -H "Content-Type: application/json" \
+-d '{ "user": "1", "num": 4 }' http://localhost:8000/queries.json
+```
+
+## Advanced Topics
+
+### Run with Elasticsearch
+
+For Elasticsearch, Meta and Event storage are available.
+To start PredictionIO with Elasticsearch,
+
+```
+docker-compose -f docker-compose.yml \
+  -f elasticsearch/docker-compose.base.yml \
+  -f elasticsearch/docker-compose.meta.yml \
+  -f elasticsearch/docker-compose.event.yml \
+  -f localfs/docker-compose.model.yml \
+  up
+```
+
+### Run with Spark Cluster
+
+Adding `docker-compose.spark.yml`, you can use Spark cluster on `pio train`.
+
+```
+docker-compose -f docker-compose.yml \
+  -f docker-compose.spark.yml \
+  -f elasticsearch/docker-compose.base.yml \
+  -f elasticsearch/docker-compose.meta.yml \
+  -f elasticsearch/docker-compose.event.yml \
+  -f localfs/docker-compose.model.yml \
+  up
+```
+
+To submit a training task to Spark Cluster, run `pio-deploy train` with `--master` option:
+
+```
+pio-docker train -- --master spark://spark-master:7077
+```
+
+See `docker-compose.spark.yml` if changing settings for Spark Cluster.
+
+### Run Engine Server
+
+To deploy your engine and start an engine server, run Docker with `docker-compose.deploy.yml`.
+
+```
+docker-compose -f docker-compose.yml \
+  -f pgsql/docker-compose.base.yml \
+  -f pgsql/docker-compose.meta.yml \
+  -f pgsql/docker-compose.event.yml \
+  -f pgsql/docker-compose.model.yml \
+  -f docker-compose.deploy.yml \
+  up
+```
+
+See `deploy/run.sh` and `docker-compose.deploy.yml` if changing a deployment.
+
+
+## Development
+
+### Build Docker Image
+
+```
+docker build -t predictionio/pio pio
+```
+
+### Push Docker Image
+
+```
+docker push predictionio/pio:latest
+docker tag predictionio/pio:latest predictionio/pio:$PIO\_VERSION
+docker push predictionio/pio:$PIO\_VERSION
+```
diff --git a/docker/bin/pio-docker b/docker/bin/pio-docker
new file mode 100755
index 0000000..8046060
--- /dev/null
+++ b/docker/bin/pio-docker
@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+BASE_WORK_DIR=/templates
+CURRENT_DIR=`pwd`
+
+get_container_id() {
+  if [ x"$PIO_CONTAINER_ID" != "x" ] ; then
+    echo $PIO_CONTAINER_ID
+    return
+  fi
+  for i in `docker ps -f "name=pio" -q` ; do
+    echo $i
+    return
+  done
+}
+
+get_current_dir() {
+  if [ x"$PIO_CURRENT_DIR" != "x" ] ; then
+    echo $PIO_CURRENT_DIR
+    return
+  fi
+  D=`echo $CURRENT_DIR | sed -e "s,.*$BASE_WORK_DIR,$BASE_WORK_DIR,"`
+  if [[ $D = $BASE_WORK_DIR* ]] ; then
+    echo $D
+  else
+    echo $BASE_WORK_DIR
+  fi
+}
+
+cid=`get_container_id`
+if [ x"$cid" = "x" ] ; then
+  echo "Docker Container is not found."
+  exit 1
+fi
+
+wdir=`get_current_dir`
+
+docker exec -w $wdir -it $cid pio $@
+
diff --git a/docker/deploy/run.sh b/docker/deploy/run.sh
new file mode 100644
index 0000000..f225a62
--- /dev/null
+++ b/docker/deploy/run.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+cd /templates/$PIO_TEMPLATE_NAME
+pio deploy
diff --git a/docker/docker-compose.deploy.yml b/docker/docker-compose.deploy.yml
new file mode 100644
index 0000000..145a960
--- /dev/null
+++ b/docker/docker-compose.deploy.yml
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      - "PIO_TEMPLATE_NAME=MyRecommendation"
+      - "PIO_RUN_FILE=/deploy/run.sh"
+    volumes:
+      - ./deploy:/deploy
diff --git a/docker/docker-compose.spark.yml b/docker/docker-compose.spark.yml
new file mode 100644
index 0000000..8f3c43e
--- /dev/null
+++ b/docker/docker-compose.spark.yml
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  spark-master:
+    image: bde2020/spark-master:2.2.2-hadoop2.7
+    container_name: spark-master
+    ports:
+      - "8080:8080"
+      - "7077:7077"
+    environment:
+      - INIT_DAEMON_STEP=setup_spark
+  spark-worker-1:
+    image: bde2020/spark-worker:2.2.2-hadoop2.7
+    container_name: spark-worker-1
+    depends_on:
+      - spark-master
+    ports:
+      - "8081:8081"
+    environment:
+      - "SPARK_MASTER=spark://spark-master:7077"
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
new file mode 100644
index 0000000..e42c907
--- /dev/null
+++ b/docker/docker-compose.yml
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    image: predictionio/pio:latest
+    ports:
+      - 7070:7070
+      - 8000:8000
+    volumes:
+      - ./templates:/templates
+    dns: 8.8.8.8
diff --git a/docker/elasticsearch/docker-compose.base.yml b/docker/elasticsearch/docker-compose.base.yml
new file mode 100644
index 0000000..4784f4a
--- /dev/null
+++ b/docker/elasticsearch/docker-compose.base.yml
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.4
+    environment:
+      - xpack.graph.enabled=false
+      - xpack.ml.enabled=false
+      - xpack.monitoring.enabled=false
+      - xpack.security.enabled=false
+      - xpack.watcher.enabled=false
+      - cluster.name=predictionio
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+  pio:
+    depends_on:
+      - elasticsearch
+    environment:
+      PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE: elasticsearch
+      PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS: elasticsearch
+      PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS: 9200
+      PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES: http
+
diff --git a/docker/elasticsearch/docker-compose.event.yml b/docker/elasticsearch/docker-compose.event.yml
new file mode 100644
index 0000000..5a77b6a
--- /dev/null
+++ b/docker/elasticsearch/docker-compose.event.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME: pio_event
+      PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE: ELASTICSEARCH
+
diff --git a/docker/elasticsearch/docker-compose.meta.yml b/docker/elasticsearch/docker-compose.meta.yml
new file mode 100644
index 0000000..0ce31dd
--- /dev/null
+++ b/docker/elasticsearch/docker-compose.meta.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      PIO_STORAGE_REPOSITORIES_METADATA_NAME: pio_meta
+      PIO_STORAGE_REPOSITORIES_METADATA_SOURCE: ELASTICSEARCH
+
diff --git a/docker/localfs/docker-compose.model.yml b/docker/localfs/docker-compose.model.yml
new file mode 100644
index 0000000..f38ff53
--- /dev/null
+++ b/docker/localfs/docker-compose.model.yml
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      PIO_STORAGE_REPOSITORIES_MODELDATA_NAME: pio_model
+      PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE: LOCALFS
+      PIO_FS_BASEDIR: /work/pio_store
+      PIO_FS_ENGINESDIR: /work/pio_store/engines
+      PIO_FS_TMPDIR: /work/pio_store/tmp
+      PIO_STORAGE_SOURCES_LOCALFS_TYPE: localfs
+      PIO_STORAGE_SOURCES_LOCALFS_PATH: /work/pio_store/models
+
diff --git a/docker/mysql/docker-compose.base.yml b/docker/mysql/docker-compose.base.yml
new file mode 100644
index 0000000..fad5309
--- /dev/null
+++ b/docker/mysql/docker-compose.base.yml
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  mysql:
+    image: mysql:8
+    command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
+    environment:
+      MYSQL_ROOT_PASSWORD: root
+      MYSQL_USER: pio
+      MYSQL_PASSWORD: pio
+      MYSQL_DATABASE: pio
+  pio:
+    depends_on:
+      - mysql
+    environment:
+      PIO_STORAGE_SOURCES_MYSQL_TYPE: jdbc
+      PIO_STORAGE_SOURCES_MYSQL_URL: "jdbc:mysql://mysql/pio"
+      PIO_STORAGE_SOURCES_MYSQL_USERNAME: pio
+      PIO_STORAGE_SOURCES_MYSQL_PASSWORD: pio
+
diff --git a/docker/mysql/docker-compose.event.yml b/docker/mysql/docker-compose.event.yml
new file mode 100644
index 0000000..f5f4035
--- /dev/null
+++ b/docker/mysql/docker-compose.event.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME: pio_event
+      PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE: MYSQL
+
diff --git a/docker/mysql/docker-compose.meta.yml b/docker/mysql/docker-compose.meta.yml
new file mode 100644
index 0000000..f7a5ece
--- /dev/null
+++ b/docker/mysql/docker-compose.meta.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      PIO_STORAGE_REPOSITORIES_METADATA_NAME: pio_meta
+      PIO_STORAGE_REPOSITORIES_METADATA_SOURCE: MYSQL
+
diff --git a/docker/mysql/docker-compose.model.yml b/docker/mysql/docker-compose.model.yml
new file mode 100644
index 0000000..4a48684
--- /dev/null
+++ b/docker/mysql/docker-compose.model.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      PIO_STORAGE_REPOSITORIES_MODELDATA_NAME: pio_model
+      PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE: MYSQL
+
diff --git a/docker/pgsql/docker-compose.base.yml b/docker/pgsql/docker-compose.base.yml
new file mode 100644
index 0000000..297d6a8
--- /dev/null
+++ b/docker/pgsql/docker-compose.base.yml
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  postgres:
+    image: postgres:9
+    environment:
+      POSTGRES_USER: pio
+      POSTGRES_PASSWORD: pio
+      POSTGRES_INITDB_ARGS: --encoding=UTF8
+  pio:
+    depends_on:
+      - postgres
+    environment:
+      PIO_STORAGE_SOURCES_PGSQL_TYPE: jdbc
+      PIO_STORAGE_SOURCES_PGSQL_URL: "jdbc:postgresql://postgres/pio"
+      PIO_STORAGE_SOURCES_PGSQL_USERNAME: pio
+      PIO_STORAGE_SOURCES_PGSQL_PASSWORD: pio
+
diff --git a/docker/pgsql/docker-compose.event.yml b/docker/pgsql/docker-compose.event.yml
new file mode 100644
index 0000000..2579141
--- /dev/null
+++ b/docker/pgsql/docker-compose.event.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME: pio_event
+      PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE: PGSQL
+
diff --git a/docker/pgsql/docker-compose.meta.yml b/docker/pgsql/docker-compose.meta.yml
new file mode 100644
index 0000000..345bfbe
--- /dev/null
+++ b/docker/pgsql/docker-compose.meta.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      PIO_STORAGE_REPOSITORIES_METADATA_NAME: pio_meta
+      PIO_STORAGE_REPOSITORIES_METADATA_SOURCE: PGSQL
+
diff --git a/docker/pgsql/docker-compose.model.yml b/docker/pgsql/docker-compose.model.yml
new file mode 100644
index 0000000..329649f
--- /dev/null
+++ b/docker/pgsql/docker-compose.model.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+  pio:
+    environment:
+      PIO_STORAGE_REPOSITORIES_MODELDATA_NAME: pio_model
+      PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE: PGSQL
+
diff --git a/docker/pio/Dockerfile b/docker/pio/Dockerfile
new file mode 100644
index 0000000..88930bd
--- /dev/null
+++ b/docker/pio/Dockerfile
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8
+
+ARG PIO_GIT_URL=https://github.com/apache/predictionio.git
+ARG PIO_TAG=v0.13.0
+ENV SCALA_VERSION=2.11.12
+ENV SPARK_VERSION=2.2.2
+ENV HADOOP_VERSION=2.7.7
+ENV ELASTICSEARCH_VERSION=5.5.3
+ENV PGSQL_VERSION=42.2.4
+ENV MYSQL_VERSION=8.0.12
+ENV PIO_HOME=/usr/share/predictionio
+
+RUN apt-get update && \
+    apt-get install -y dpkg-dev fakeroot && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /opt/src
+RUN git clone -b $PIO_TAG $PIO_GIT_URL
+WORKDIR /opt/src/predictionio
+RUN bash ./make-distribution.sh \
+      -Dscala.version=$SCALA_VERSION \
+      -Dspark.version=$SPARK_VERSION \
+      -Dhadoop.version=$HADOOP_VERSION \
+      -Delasticsearch.version=$ELASTICSEARCH_VERSION \
+      --with-deb && \
+    dpkg -i ./assembly/target/predictionio_*.deb && \
+    rm -rf /opt/src/predictionio/*
+
+
+RUN cp /etc/predictionio/pio-env.sh /etc/predictionio/pio-env.sh.orig && \
+    echo "#!/usr/bin/env bash" > /etc/predictionio/pio-env.sh
+RUN curl -o $PIO_HOME/lib/postgresql-$PGSQL_VERSION.jar \
+    http://central.maven.org/maven2/org/postgresql/postgresql/$PGSQL_VERSION/postgresql-$PGSQL_VERSION.jar
&& \
+    echo "POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-$PGSQL_VERSION.jar" >> /etc/predictionio/pio-env.sh
&& \
+    echo "MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-$MYSQL_VERSION.jar" >>
/etc/predictionio/pio-env.sh
+
+WORKDIR /usr/share
+RUN curl -o /opt/src/spark-$SPARK_VERSION.tgz \
+    http://www-us.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop2.7.tgz
&& \
+    tar zxvf /opt/src/spark-$SPARK_VERSION.tgz && \
+    echo "SPARK_HOME="`pwd`/`ls -d spark*` >> /etc/predictionio/pio-env.sh &&
\
+    rm -rf /opt/src
+
+WORKDIR /templates
+ADD pio_run /usr/bin/pio_run
+
+EXPOSE 7070
+EXPOSE 8000
+
+CMD ["sh", "/usr/bin/pio_run"]
+
diff --git a/docker/pio/pio_run b/docker/pio/pio_run
new file mode 100644
index 0000000..83ac6cc
--- /dev/null
+++ b/docker/pio/pio_run
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+. /etc/predictionio/pio-env.sh
+
+# check elasticsearch status
+if [ x"$PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE" != "x" ] ; then
+  RET=-1
+  COUNT=0
+  ES_HOST=`echo $PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS | sed -e "s/,.*//"`
+  ES_PORT=`echo $PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS | sed -e "s/,.*//"`
+  # Wait for elasticsearch startup
+  while [ $RET != 0 -a $COUNT -lt 10 ] ; do
+    echo "Waiting for ${ES_HOST}..."
+    curl --connect-timeout 60 --retry 10 -s "$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=green&timeout=1m"
+    RET=$?
+    COUNT=`expr $COUNT + 1`
+    sleep 1
+  done
+fi
+
+# check mysql jar file
+if [ x"$PIO_STORAGE_SOURCES_MYSQL_TYPE" != "x" ] ; then
+  MYSQL_JAR_FILE=$PIO_HOME/lib/mysql-connector-java-$MYSQL_VERSION.jar
+  if [ ! -f $MYSQL_JAR_FILE ] ; then
+    curl -o $MYSQL_JAR_FILE http://central.maven.org/maven2/mysql/mysql-connector-java/$MYSQL_VERSION/mysql-connector-java-$MYSQL_VERSION.jar
+  fi
+fi
+
+# Check PIO status
+RET=-1
+COUNT=0
+while [ $RET != 0 -a $COUNT -lt 10 ] ; do
+  echo "Waiting for PredictionIO..."
+  $PIO_HOME/bin/pio status
+  RET=$?
+  COUNT=`expr $COUNT + 1`
+  sleep 1
+done
+
+
+if [ x"$PIO_RUN_FILE" != "x" ] ; then
+  sh $PIO_RUN_FILE
+else
+  # Start PIO Event Server
+  $PIO_HOME/bin/pio eventserver
+fi
+
diff --git a/docker/templates/.keep b/docker/templates/.keep
new file mode 100644
index 0000000..ec20143
--- /dev/null
+++ b/docker/templates/.keep
@@ -0,0 +1,14 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.


Mime
View raw message