predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shinsuke Sugaya <shinsuke.sug...@gmail.com>
Subject Re: Unable to connect to all storage backends successfully
Date Sun, 24 Sep 2017 23:29:19 GMT
Hi,

> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300

Elasticsearch 5 uses 9200 port.
Could you try 9200 port?

Thanks,
 shinsuke


2017-09-24 3:02 GMT+09:00 Jim Miller <jemiller1963@gmail.com>:
> Hi Donald,
>
> Tried just now and received the following error:
>
> vagrant:~/ $ pio status
> [13:34:52]
> [INFO] [Management$] Inspecting PredictionIO...
> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating
> [INFO] [Management$] Inspecting Apache Spark...
> [INFO] [Management$] Apache Spark is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.7
> [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum requirement
> of 1.3.0)
> [INFO] [Management$] Inspecting storage backend connections...
> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
> [ERROR] [Management$] Unable to connect to all storage backends
> successfully.
> The following shows the error message from the storage backend.
>
> error while performing request (java.lang.RuntimeException)
>
> Dumping configuration of initialized storage backend sources.
> Please make sure they are correct.
>
> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.5.2,
> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE ->
> elastic search
>
>
> HERE IS MY PIO-ENV.SH
>
> # PredictionIO Main Configuration
> #
> # This section controls core behavior of PredictionIO. It is very likely
> that
> # you need to change these to fit your site.
>
> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
> SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
>
> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>
> # ES_CONF_DIR: You must configure this if you have advanced configuration
> for
> #              your Elasticsearch setup.
> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2
> # HADOOP_CONF_DIR=/opt/hadoop
>
> # HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
> #                 with HBase on a remote cluster.
> HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf
>
> # Filesystem paths where PredictionIO uses as block storage.
> PIO_FS_BASEDIR=$HOME/.pio_store
> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>
> # PredictionIO Storage Configuration
> #
> # This section controls programs that make use of PredictionIO's built-in
> # storage facilities. Default values are shown below.
> #
> # For more information on storage configuration please refer to
> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>
> # Storage Repositories
>
> # Default is to use PostgreSQL
> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>
> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>
> # Storage Data Sources
>
> # PostgreSQL Default Settings
> # Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>
> # MySQL Example
> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>
> # Elasticsearch Example
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
> #
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
> # Optional basic HTTP auth
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
> # Elasticsearch 1.x Example
> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
>
> # Local File System Example
> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>
> # HBase Example
> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1
>
> # AWS S3 Example
> # PIO_STORAGE_SOURCES_S3_TYPE=s3
> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>
>
> ELASTICSEARCH.YML
>
> #======================== Elasticsearch Configuration
> =========================
> #
> # NOTE: Elasticsearch comes with reasonable defaults for most settings.
> #       Before you set out to tweak and tune the configuration, make sure
> you
> #       understand what are you trying to accomplish and the consequences.
> #
> # The primary way of configuring a node is via this file. This template
> lists
> # the most important settings you may want to configure for a production
> cluster.
> #
> # Please consult the documentation for further information on configuration
> options:
> # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
> #
> # ---------------------------------- Cluster
> -----------------------------------
> #
> # Use a descriptive name for your cluster:
> #
> cluster.name: firstCluster
> #
> # ------------------------------------ Node
> ------------------------------------
> #
> # Use a descriptive name for the node:
> #
> #node.name: node-1
> #
> # Add custom attributes to the node:
> #
> #node.attr.rack: r1
> #
> # ----------------------------------- Paths
> ------------------------------------
> #
> # Path to directory where to store the data (separate multiple locations by
> comma):
> #
> #path.data: /path/to/data
> #
> # Path to log files:
> #
> #path.logs: /path/to/logs
> #
> # ----------------------------------- Memory
> -----------------------------------
> #
> # Lock the memory on startup:
> #
> #bootstrap.memory_lock: true
> #
> # Make sure that the heap size is set to about half the memory available
> # on the system and that the owner of the process is allowed to use this
> # limit.
> #
> # Elasticsearch performs poorly when the system is swapping the memory.
> #
> # ---------------------------------- Network
> -----------------------------------
> #
> # Set the bind address to a specific IP (IPv4 or IPv6):
> #
> #network.host: 192.168.0.1
> #
> # Set a custom port for HTTP:
> #
> #http.port: 9200
> #
> # For more information, consult the network module documentation.
> #
> # --------------------------------- Discovery
> ----------------------------------
> #
> # Pass an initial list of hosts to perform discovery when new node is
> started:
> # The default list of hosts is ["127.0.0.1", "[::1]"]
> #
> #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
> #
> # Prevent the "split brain" by configuring the majority of nodes (total
> number of master-eligible nodes / 2 + 1):
> #
> #discovery.zen.minimum_master_nodes: 3
> #
> # For more information, consult the zen discovery module documentation.
> #
> # ---------------------------------- Gateway
> -----------------------------------
> #
> # Block initial recovery after a full cluster restart until N nodes are
> started:
> #
> #gateway.recover_after_nodes: 3
> #
> # For more information, consult the gateway module documentation.
> #
> # ---------------------------------- Various
> -----------------------------------
> #
> # Require explicit names when deleting indices:
> #
> #action.destructive_requires_name: true
>
>
> Thanks for all of your help!
>
> Jim
> --
> Jim Miller
>
> On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com)
> wrote:
>
> Hi Donald,
>
> I did not.  I will read the release notes and update accordingly.
>
> Thanks!
>
> Jim
>
> --
> Jim Miller
>
> On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org) wrote:
>
> Hey Jim,
>
> Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in
> 0.12 so the default build will use ES 5.x support.
>
> See the upcoming release notes:
> https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes
>
> Regards,
> Donald
>
> On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <jemiller1963@gmail.com> wrote:
>>
>> Yes.  I’m following this tutorial but using 0.12.0 instead of 0.10.0:
>>
>> https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
>>
>> --
>> Jim Miller
>>
>> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com)
>> wrote:
>>
>> meaning is “firstcluster” the cluster name in your Elasticsearch
>> configuration?
>>
>>
>> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <vaghawan781@gmail.com> wrote:
>>
>> I think the problem is with Elasticsearch, are you sure the cluster exists
>> in elasticsearch configuration?
>>
>> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <jemiller1963@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
>>>
>>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
>>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
>>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>>
>>> All starts with no errors but with pio status I get:
>>>
>>> [INFO] [Management$] Inspecting PredictionIO...
>>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
>>> [INFO] [Management$] Inspecting Apache Spark...
>>> [INFO] [Management$] Apache Spark is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
>>> requirement of 1.3.0)
>>> [INFO] [Management$] Inspecting storage backend connections...
>>> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
>>> [ERROR] [Management$] Unable to connect to all storage backends
>>> successfully.
>>> The following shows the error message from the storage backend.
>>>
>>> Connection closed
>>> (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
>>>
>>> Dumping configuration of initialized storage backend sources.
>>> Please make sure they are correct.
>>>
>>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4,
>>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE
->
>>> elasticsearch
>>>
>>>
>>> Can anyone give me an idea of what I need to fix this issue?  Here is
>>>
>>>
>>> # PredictionIO Main Configuration
>>> #
>>> # This section controls core behavior of PredictionIO. It is very likely
>>> that
>>> # you need to change these to fit your site.
>>>
>>> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
>>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>>>
>>> # ES_CONF_DIR: You must configure this if you have advanced configuration
>>> for
>>> #              your Elasticsearch setup.
>>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>>>
>>> # HADOOP_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> #                  with Hadoop 2.
>>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>>>
>>> # HBASE_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> #                 with HBase on a remote cluster.
>>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
>>>
>>> # Filesystem paths where PredictionIO uses as block storage.
>>> PIO_FS_BASEDIR=$HOME/.pio_store
>>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>>>
>>> # PredictionIO Storage Configuration
>>> #
>>> # This section controls programs that make use of PredictionIO's built-in
>>> # storage facilities. Default values are shown below.
>>> #
>>> # For more information on storage configuration please refer to
>>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>>>
>>> # Storage Repositories
>>>
>>> # Default is to use PostgreSQL
>>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>>>
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>>>
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>>>
>>> # Storage Data Sources
>>>
>>> # PostgreSQL Default Settings
>>> # Please change "pio" to your database name in
>>> PIO_STORAGE_SOURCES_PGSQL_URL
>>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>>>
>>> # MySQL Example
>>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>>>
>>> # Elasticsearch Example
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>>> #
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
>>> # Optional basic HTTP auth
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>>> # Elasticsearch 1.x Example
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>>
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
>>>
>>> # Local File System Example
>>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>>>
>>> # HBase Example
>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
>>>
>>> # AWS S3 Example
>>> # PIO_STORAGE_SOURCES_S3_TYPE=s3
>>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>>>
>>>
>>> Thanks,
>>>
>>> Jim
>>>
>>>
>>> --
>>> Jim Miller
>>
>>
>>
>

Mime
View raw message