Mailing-List: contact user-help@predictionio.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@predictionio.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <etPan.59c6a1a3.234a5e9b.1e2@Tesla.local>
References: <etPan.59c1d343.5d35d0e.1e2@Tesla.local> <CA+69RXbCZpLgOcBG4wKDL5ComD9Hbhzx8n0Y181RuffHgNiCog@mail.gmail.com>
 <DD5B3D4C-B732-441B-A8A2-00B826DBFC9D@occamsmachete.com> <etPan.59c285ef.21ef2bb4.1e2@Tesla.local>
 <CAD8z1JKq87caitaRNf2QebkmmP=f3nrR9nZ7q3EmvYcBgktwQA@mail.gmail.com>
 <etPan.59c3127c.e612e14.1e2@Tesla.local> <etPan.59c6a1a3.234a5e9b.1e2@Tesla.local>
From: Shinsuke Sugaya <shinsuke.sugaya@gmail.com>
Date: Mon, 25 Sep 2017 08:29:19 +0900
Message-ID: <CA+0WS37CLCOhQeiF=J1Xo0XNGAxyQk=RLGTXzWawGR0oZvnCAg@mail.gmail.com>
Subject: Re: Unable to connect to all storage backends successfully
To: user@predictionio.incubator.apache.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
archived-at: Sun, 24 Sep 2017 23:29:34 -0000

Hi,

> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=3D9300

Elasticsearch 5 uses 9200 port.
Could you try 9200 port?

Thanks,
 shinsuke


2017-09-24 3:02 GMT+09:00 Jim Miller <jemiller1963@gmail.com>:
> Hi Donald,
>
> Tried just now and received the following error:
>
> vagrant:~/ $ pio status
> [13:34:52]
> [INFO] [Management$] Inspecting PredictionIO...
> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating
> [INFO] [Management$] Inspecting Apache Spark...
> [INFO] [Management$] Apache Spark is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-=
hadoop2.7
> [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum requireme=
nt
> of 1.3.0)
> [INFO] [Management$] Inspecting storage backend connections...
> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
> [ERROR] [Management$] Unable to connect to all storage backends
> successfully.
> The following shows the error message from the storage backend.
>
> error while performing request (java.lang.RuntimeException)
>
> Dumping configuration of initialized storage backend sources.
> Please make sure they are correct.
>
> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.=
5.2,
> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE ->
> elastic search
>
>
> HERE IS MY PIO-ENV.SH
>
> # PredictionIO Main Configuration
> #
> # This section controls core behavior of PredictionIO. It is very likely
> that
> # you need to change these to fit your site.
>
> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
> # SPARK_HOME=3D$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
> SPARK_HOME=3D$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
>
> POSTGRES_JDBC_DRIVER=3D$PIO_HOME/lib/postgresql-42.0.0.jar
> MYSQL_JDBC_DRIVER=3D$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>
> # ES_CONF_DIR: You must configure this if you have advanced configuration
> for
> #              your Elasticsearch setup.
> ES_CONF_DIR=3D$PIO_HOME/vendors/elasticsearch-5.5.2
> # HADOOP_CONF_DIR=3D/opt/hadoop
>
> # HBASE_CONF_DIR: You must configure this if you intend to run Prediction=
IO
> #                 with HBase on a remote cluster.
> HBASE_CONF_DIR=3D$PIO_HOME/hbase-1.3.1/conf
>
> # Filesystem paths where PredictionIO uses as block storage.
> PIO_FS_BASEDIR=3D$HOME/.pio_store
> PIO_FS_ENGINESDIR=3D$PIO_FS_BASEDIR/engines
> PIO_FS_TMPDIR=3D$PIO_FS_BASEDIR/tmp
>
> # PredictionIO Storage Configuration
> #
> # This section controls programs that make use of PredictionIO's built-in
> # storage facilities. Default values are shown below.
> #
> # For more information on storage configuration please refer to
> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>
> # Storage Repositories
>
> # Default is to use PostgreSQL
> PIO_STORAGE_REPOSITORIES_METADATA_NAME=3Dpio_meta
> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=3DELASTICSEARCH
>
> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=3Dpio_event
> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=3DHBASE
>
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=3Dpio_model
> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=3DLOCALFS
>
> # Storage Data Sources
>
> # PostgreSQL Default Settings
> # Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_=
URL
> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
> # PIO_STORAGE_SOURCES_PGSQL_TYPE=3Djdbc
> # PIO_STORAGE_SOURCES_PGSQL_URL=3Djdbc:postgresql://localhost/pio
> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=3Dpio
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=3Dpio
>
> # MySQL Example
> # PIO_STORAGE_SOURCES_MYSQL_TYPE=3Djdbc
> # PIO_STORAGE_SOURCES_MYSQL_URL=3Djdbc:mysql://localhost/pio
> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=3Dpio
> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=3Dpio
>
> # Elasticsearch Example
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=3Delasticsearch
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=3Dlocalhost
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=3D9200
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=3Dhttp
> #
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=3D$PIO_HOME/vendors/elasticsearch-=
5.5.2
> # Optional basic HTTP auth
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=3Dmy-name
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=3Dmy-secret
> # Elasticsearch 1.x Example
> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=3Delasticsearch
> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=3DfirstCluster
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=3Dlocalhost
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=3D9300
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=3D$PIO_HOME/vendors/elasticsearch-=
5.5.2
>
> # Local File System Example
> PIO_STORAGE_SOURCES_LOCALFS_TYPE=3Dlocalfs
> PIO_STORAGE_SOURCES_LOCALFS_PATH=3D$PIO_FS_BASEDIR/models
>
> # HBase Example
> PIO_STORAGE_SOURCES_HBASE_TYPE=3Dhbase
> PIO_STORAGE_SOURCES_HBASE_HOME=3D$PIO_HOME/vendors/hbase-1.3.1
>
> # AWS S3 Example
> # PIO_STORAGE_SOURCES_S3_TYPE=3Ds3
> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=3Dpio_bucket
> # PIO_STORAGE_SOURCES_S3_BASE_PATH=3Dpio_model
>
>
> ELASTICSEARCH.YML
>
> #=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 Elasticsearch Configuration
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
> #
> # NOTE: Elasticsearch comes with reasonable defaults for most settings.
> #       Before you set out to tweak and tune the configuration, make sure
> you
> #       understand what are you trying to accomplish and the consequences=
.
> #
> # The primary way of configuring a node is via this file. This template
> lists
> # the most important settings you may want to configure for a production
> cluster.
> #
> # Please consult the documentation for further information on configurati=
on
> options:
> # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
> #
> # ---------------------------------- Cluster
> -----------------------------------
> #
> # Use a descriptive name for your cluster:
> #
> cluster.name: firstCluster
> #
> # ------------------------------------ Node
> ------------------------------------
> #
> # Use a descriptive name for the node:
> #
> #node.name: node-1
> #
> # Add custom attributes to the node:
> #
> #node.attr.rack: r1
> #
> # ----------------------------------- Paths
> ------------------------------------
> #
> # Path to directory where to store the data (separate multiple locations =
by
> comma):
> #
> #path.data: /path/to/data
> #
> # Path to log files:
> #
> #path.logs: /path/to/logs
> #
> # ----------------------------------- Memory
> -----------------------------------
> #
> # Lock the memory on startup:
> #
> #bootstrap.memory_lock: true
> #
> # Make sure that the heap size is set to about half the memory available
> # on the system and that the owner of the process is allowed to use this
> # limit.
> #
> # Elasticsearch performs poorly when the system is swapping the memory.
> #
> # ---------------------------------- Network
> -----------------------------------
> #
> # Set the bind address to a specific IP (IPv4 or IPv6):
> #
> #network.host: 192.168.0.1
> #
> # Set a custom port for HTTP:
> #
> #http.port: 9200
> #
> # For more information, consult the network module documentation.
> #
> # --------------------------------- Discovery
> ----------------------------------
> #
> # Pass an initial list of hosts to perform discovery when new node is
> started:
> # The default list of hosts is ["127.0.0.1", "[::1]"]
> #
> #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
> #
> # Prevent the "split brain" by configuring the majority of nodes (total
> number of master-eligible nodes / 2 + 1):
> #
> #discovery.zen.minimum_master_nodes: 3
> #
> # For more information, consult the zen discovery module documentation.
> #
> # ---------------------------------- Gateway
> -----------------------------------
> #
> # Block initial recovery after a full cluster restart until N nodes are
> started:
> #
> #gateway.recover_after_nodes: 3
> #
> # For more information, consult the gateway module documentation.
> #
> # ---------------------------------- Various
> -----------------------------------
> #
> # Require explicit names when deleting indices:
> #
> #action.destructive_requires_name: true
>
>
> Thanks for all of your help!
>
> Jim
> --
> Jim Miller
>
> On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com)
> wrote:
>
> Hi Donald,
>
> I did not.  I will read the release notes and update accordingly.
>
> Thanks!
>
> Jim
>
> --
> Jim Miller
>
> On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org) wro=
te:
>
> Hey Jim,
>
> Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in
> 0.12 so the default build will use ES 5.x support.
>
> See the upcoming release notes:
> https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELE=
ASE.md#behavior-changes
>
> Regards,
> Donald
>
> On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <jemiller1963@gmail.com> wrot=
e:
>>
>> Yes.  I=E2=80=99m following this tutorial but using 0.12.0 instead of 0.=
10.0:
>>
>> https://medium.freecodecamp.org/building-an-recommendation-engine-with-a=
pache-prediction-io-ml-server-aed0319e0d8
>>
>> --
>> Jim Miller
>>
>> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com)
>> wrote:
>>
>> meaning is =E2=80=9Cfirstcluster=E2=80=9D the cluster name in your Elast=
icsearch
>> configuration?
>>
>>
>> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <vaghawan781@gmail.com> wrote=
:
>>
>> I think the problem is with Elasticsearch, are you sure the cluster exis=
ts
>> in elasticsearch configuration?
>>
>> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <jemiller1963@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I=E2=80=99m using PredictionIO 0.12.0-incubating with ElasticSearch and=
 Hbase:
>>>
>>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
>>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
>>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>>
>>> All starts with no errors but with pio status I get:
>>>
>>> [INFO] [Management$] Inspecting PredictionIO...
>>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
>>> [INFO] [Management$] Inspecting Apache Spark...
>>> [INFO] [Management$] Apache Spark is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bi=
n-hadoop2.6
>>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
>>> requirement of 1.3.0)
>>> [INFO] [Management$] Inspecting storage backend connections...
>>> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)..=
.
>>> [ERROR] [Management$] Unable to connect to all storage backends
>>> successfully.
>>> The following shows the error message from the storage backend.
>>>
>>> Connection closed
>>> (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedExcepti=
on)
>>>
>>> Dumping configuration of initialized storage backend sources.
>>> Please make sure they are correct.
>>>
>>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-=
1.4.4,
>>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE ->
>>> elasticsearch
>>>
>>>
>>> Can anyone give me an idea of what I need to fix this issue?  Here is
>>>
>>>
>>> # PredictionIO Main Configuration
>>> #
>>> # This section controls core behavior of PredictionIO. It is very likel=
y
>>> that
>>> # you need to change these to fit your site.
>>>
>>> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
>>> # SPARK_HOME=3D$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>>> SPARK_HOME=3D$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>> POSTGRES_JDBC_DRIVER=3D$PIO_HOME/lib/postgresql-42.0.0.jar
>>> MYSQL_JDBC_DRIVER=3D$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>>>
>>> # ES_CONF_DIR: You must configure this if you have advanced configurati=
on
>>> for
>>> #              your Elasticsearch setup.
>>> ES_CONF_DIR=3D$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>>>
>>> # HADOOP_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> #                  with Hadoop 2.
>>> HADOOP_CONF_DIR=3D$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>>>
>>> # HBASE_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> #                 with HBase on a remote cluster.
>>> HBASE_CONF_DIR=3D$PIO_HOME/vendors/hbase-1.0.0/conf
>>>
>>> # Filesystem paths where PredictionIO uses as block storage.
>>> PIO_FS_BASEDIR=3D$HOME/.pio_store
>>> PIO_FS_ENGINESDIR=3D$PIO_FS_BASEDIR/engines
>>> PIO_FS_TMPDIR=3D$PIO_FS_BASEDIR/tmp
>>>
>>> # PredictionIO Storage Configuration
>>> #
>>> # This section controls programs that make use of PredictionIO's built-=
in
>>> # storage facilities. Default values are shown below.
>>> #
>>> # For more information on storage configuration please refer to
>>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>>>
>>> # Storage Repositories
>>>
>>> # Default is to use PostgreSQL
>>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=3Dpio_meta
>>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=3DELASTICSEARCH
>>>
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=3Dpio_event
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=3DHBASE
>>>
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=3Dpio_model
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=3DLOCALFS
>>>
>>> # Storage Data Sources
>>>
>>> # PostgreSQL Default Settings
>>> # Please change "pio" to your database name in
>>> PIO_STORAGE_SOURCES_PGSQL_URL
>>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=3Djdbc
>>> # PIO_STORAGE_SOURCES_PGSQL_URL=3Djdbc:postgresql://localhost/pio
>>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=3Dpio
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=3Dpio
>>>
>>> # MySQL Example
>>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=3Djdbc
>>> # PIO_STORAGE_SOURCES_MYSQL_URL=3Djdbc:mysql://localhost/pio
>>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=3Dpio
>>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=3Dpio
>>>
>>> # Elasticsearch Example
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=3Delasticsearch
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=3Dlocalhost
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=3D9200
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=3Dhttp
>>> #
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=3D$PIO_HOME/vendors/elasticsearc=
h-5.5.2
>>> # Optional basic HTTP auth
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=3Dmy-name
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=3Dmy-secret
>>> # Elasticsearch 1.x Example
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=3Delasticsearch
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=3Dfirstcluster
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=3Dlocalhost
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=3D9300
>>>
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=3D$PIO_HOME/vendors/elasticsearc=
h-1.4.4
>>>
>>> # Local File System Example
>>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=3Dlocalfs
>>> PIO_STORAGE_SOURCES_LOCALFS_PATH=3D$PIO_FS_BASEDIR/models
>>>
>>> # HBase Example
>>> PIO_STORAGE_SOURCES_HBASE_TYPE=3Dhbase
>>> PIO_STORAGE_SOURCES_HBASE_HOME=3D$PIO_HOME/vendors/hbase-1.0.0
>>>
>>> # AWS S3 Example
>>> # PIO_STORAGE_SOURCES_S3_TYPE=3Ds3
>>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=3Dpio_bucket
>>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=3Dpio_model
>>>
>>>
>>> Thanks,
>>>
>>> Jim
>>>
>>>
>>> --
>>> Jim Miller
>>
>>
>>
>