predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Miller <jemiller1...@gmail.com>
Subject Re: Unable to connect to all storage backends successfully
Date Wed, 27 Sep 2017 19:49:13 GMT
incubator-predictionio-template-recommender


On Wed, Sep 27, 2017 at 2:25 PM, Donald Szeto <donald@apache.org> wrote:

> What template and version are you using?
>
> On Tue, Sep 26, 2017 at 9:28 PM Jim Miller <jemiller1963@gmail.com> wrote:
>
>> Hi guys,
>>
>> I have another error that I cannot get past:
>>
>> *vagrant:MyRecommendation/ (develop*) $ pio train
>>
>>                                                                  [21:47:39]*
>> *[ERROR] [Console$] unable to read engine factory from
>> /home/vagrant/MyRecommendation/engine.json. Aborting.*
>>
>>
>> Does anyone have any advice?
>>
>> Thanks,
>>
>> Jim
>>
>> --
>> Jim Miller
>>
>> On September 26, 2017 at 7:42:02 PM, Jim Miller (jemiller1963@gmail.com)
>> wrote:
>>
>> Shinsuke, you rock!  That was it.
>>
>> Thank you so much,
>>
>> Jim
>>
>> --
>> Jim Miller
>>
>> On September 24, 2017 at 7:29:28 PM, Shinsuke Sugaya (
>> shinsuke.sugaya@gmail.com) wrote:
>>
>> Hi,
>>
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>
>> Elasticsearch 5 uses 9200 port.
>> Could you try 9200 port?
>>
>> Thanks,
>> shinsuke
>>
>>
>> 2017-09-24 3:02 GMT+09:00 Jim Miller <jemiller1963@gmail.com>:
>> > Hi Donald,
>> >
>> > Tried just now and received the following error:
>> >
>> > vagrant:~/ $ pio status
>> > [13:34:52]
>> > [INFO] [Management$] Inspecting PredictionIO...
>> > [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>> > /home/vagrant/pio/PredictionIO-0.12.0-incubating
>> > [INFO] [Management$] Inspecting Apache Spark...
>> > [INFO] [Management$] Apache Spark is installed at
>> > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.
>> 1-bin-hadoop2.7
>> > [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum
>> requirement
>> > of 1.3.0)
>> > [INFO] [Management$] Inspecting storage backend connections...
>> > [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
>> > [ERROR] [Management$] Unable to connect to all storage backends
>> > successfully.
>> > The following shows the error message from the storage backend.
>> >
>> > error while performing request (java.lang.RuntimeException)
>> >
>> > Dumping configuration of initialized storage backend sources.
>> > Please make sure they are correct.
>> >
>> > Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
>> > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/
>> elasticsearch-5.5.2,
>> > HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE
->
>> > elastic search
>> >
>> >
>> > HERE IS MY PIO-ENV.SH
>> >
>> > # PredictionIO Main Configuration
>> > #
>> > # This section controls core behavior of PredictionIO. It is very likely
>> > that
>> > # you need to change these to fit your site.
>> >
>> > # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
>> > # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>> > SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
>> >
>> > POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>> > MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>> >
>> > # ES_CONF_DIR: You must configure this if you have advanced
>> configuration
>> > for
>> > # your Elasticsearch setup.
>> > ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2
>> > # HADOOP_CONF_DIR=/opt/hadoop
>> >
>> > # HBASE_CONF_DIR: You must configure this if you intend to run
>> PredictionIO
>> > # with HBase on a remote cluster.
>> > HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf
>> >
>> > # Filesystem paths where PredictionIO uses as block storage.
>> > PIO_FS_BASEDIR=$HOME/.pio_store
>> > PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>> > PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>> >
>> > # PredictionIO Storage Configuration
>> > #
>> > # This section controls programs that make use of PredictionIO's
>> built-in
>> > # storage facilities. Default values are shown below.
>> > #
>> > # For more information on storage configuration please refer to
>> > # http://predictionio.incubator.apache.org/system/anotherdatastore/
>> >
>> > # Storage Repositories
>> >
>> > # Default is to use PostgreSQL
>> > PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>> > PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>> >
>> > PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>> > PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>> >
>> > PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>> > PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>> >
>> > # Storage Data Sources
>> >
>> > # PostgreSQL Default Settings
>> > # Please change "pio" to your database name in
>> PIO_STORAGE_SOURCES_PGSQL_URL
>> > # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>> > # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>> > # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>> > # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>> >
>> > # MySQL Example
>> > # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>> > # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>> > # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>> > # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>> >
>> > # Elasticsearch Example
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>> > #
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
>> vendors/elasticsearch-5.5.2
>> > # Optional basic HTTP auth
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>> > # Elasticsearch 1.x Example
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
>> vendors/elasticsearch-5.5.2
>> >
>> > # Local File System Example
>> > PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>> > PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>> >
>> > # HBase Example
>> > PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>> > PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1
>> >
>> > # AWS S3 Example
>> > # PIO_STORAGE_SOURCES_S3_TYPE=s3
>> > # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>> > # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>> >
>> >
>> > ELASTICSEARCH.YML
>> >
>> > #======================== Elasticsearch Configuration
>> > =========================
>> > #
>> > # NOTE: Elasticsearch comes with reasonable defaults for most settings.
>> > # Before you set out to tweak and tune the configuration, make sure
>> > you
>> > # understand what are you trying to accomplish and the consequences.
>> > #
>> > # The primary way of configuring a node is via this file. This template
>> > lists
>> > # the most important settings you may want to configure for a production
>> > cluster.
>> > #
>> > # Please consult the documentation for further information on
>> configuration
>> > options:
>> > # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
>> > #
>> > # ---------------------------------- Cluster
>> > -----------------------------------
>> > #
>> > # Use a descriptive name for your cluster:
>> > #
>> > cluster.name: firstCluster
>> > #
>> > # ------------------------------------ Node
>> > ------------------------------------
>> > #
>> > # Use a descriptive name for the node:
>> > #
>> > #node.name: node-1
>> > #
>> > # Add custom attributes to the node:
>> > #
>> > #node.attr.rack: r1
>> > #
>> > # ----------------------------------- Paths
>> > ------------------------------------
>> > #
>> > # Path to directory where to store the data (separate multiple
>> locations by
>> > comma):
>> > #
>> > #path.data: /path/to/data
>> > #
>> > # Path to log files:
>> > #
>> > #path.logs: /path/to/logs
>> > #
>> > # ----------------------------------- Memory
>> > -----------------------------------
>> > #
>> > # Lock the memory on startup:
>> > #
>> > #bootstrap.memory_lock: true
>> > #
>> > # Make sure that the heap size is set to about half the memory available
>> > # on the system and that the owner of the process is allowed to use this
>> > # limit.
>> > #
>> > # Elasticsearch performs poorly when the system is swapping the memory.
>> > #
>> > # ---------------------------------- Network
>> > -----------------------------------
>> > #
>> > # Set the bind address to a specific IP (IPv4 or IPv6):
>> > #
>> > #network.host: 192.168.0.1
>> > #
>> > # Set a custom port for HTTP:
>> > #
>> > #http.port: 9200
>> > #
>> > # For more information, consult the network module documentation.
>> > #
>> > # --------------------------------- Discovery
>> > ----------------------------------
>> > #
>> > # Pass an initial list of hosts to perform discovery when new node is
>> > started:
>> > # The default list of hosts is ["127.0.0.1", "[::1]"]
>> > #
>> > #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
>> > #
>> > # Prevent the "split brain" by configuring the majority of nodes (total
>> > number of master-eligible nodes / 2 + 1):
>> > #
>> > #discovery.zen.minimum_master_nodes: 3
>> > #
>> > # For more information, consult the zen discovery module documentation.
>> > #
>> > # ---------------------------------- Gateway
>> > -----------------------------------
>> > #
>> > # Block initial recovery after a full cluster restart until N nodes are
>> > started:
>> > #
>> > #gateway.recover_after_nodes: 3
>> > #
>> > # For more information, consult the gateway module documentation.
>> > #
>> > # ---------------------------------- Various
>> > -----------------------------------
>> > #
>> > # Require explicit names when deleting indices:
>> > #
>> > #action.destructive_requires_name: true
>> >
>> >
>> > Thanks for all of your help!
>> >
>> > Jim
>> > --
>> > Jim Miller
>> >
>> > On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com
>> )
>> > wrote:
>> >
>> > Hi Donald,
>> >
>> > I did not. I will read the release notes and update accordingly.
>> >
>> > Thanks!
>> >
>> > Jim
>> >
>> > --
>> > Jim Miller
>> >
>> > On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org)
>> wrote:
>> >
>> > Hey Jim,
>> >
>> > Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated
>> in
>> > 0.12 so the default build will use ES 5.x support.
>> >
>> > See the upcoming release notes:
>> > https://github.com/apache/incubator-predictionio/blob/
>> release/0.12.0/RELEASE.md#behavior-changes
>> >
>> > Regards,
>> > Donald
>> >
>> > On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <jemiller1963@gmail.com>
>> wrote:
>> >>
>> >> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
>> >>
>> >> https://medium.freecodecamp.org/building-an-
>> recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
>> >>
>> >> --
>> >> Jim Miller
>> >>
>> >> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (
>> pat@occamsmachete.com)
>> >> wrote:
>> >>
>> >> meaning is “firstcluster” the cluster name in your Elasticsearch
>> >> configuration?
>> >>
>> >>
>> >> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <vaghawan781@gmail.com>
>> wrote:
>> >>
>> >> I think the problem is with Elasticsearch, are you sure the cluster
>> exists
>> >> in elasticsearch configuration?
>> >>
>> >> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <jemiller1963@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
>> >>>
>> >>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
>> >>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
>> >>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>> >>>
>> >>>
>> >>> All starts with no errors but with pio status I get:
>> >>>
>> >>> [INFO] [Management$] Inspecting PredictionIO...
>> >>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
>> >>> [INFO] [Management$] Inspecting Apache Spark...
>> >>> [INFO] [Management$] Apache Spark is installed at
>> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.
>> 1-bin-hadoop2.6
>> >>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
>> >>> requirement of 1.3.0)
>> >>> [INFO] [Management$] Inspecting storage backend connections...
>> >>> [INFO] [Storage$] Verifying Meta Data Backend (Source:
>> ELASTICSEARCH)...
>> >>> [ERROR] [Management$] Unable to connect to all storage backends
>> >>> successfully.
>> >>> The following shows the error message from the storage backend.
>> >>>
>> >>> Connection closed
>> >>> (org.apache.predictionio.shaded.org.apache.http.
>> ConnectionClosedException)
>> >>>
>> >>> Dumping configuration of initialized storage backend sources.
>> >>> Please make sure they are correct.
>> >>>
>> >>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME
>> ->
>> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/
>> elasticsearch-1.4.4,
>> >>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster,
TYPE
>> ->
>> >>> elasticsearch
>> >>>
>> >>>
>> >>> Can anyone give me an idea of what I need to fix this issue? Here is
>> >>>
>> >>>
>> >>> # PredictionIO Main Configuration
>> >>> #
>> >>> # This section controls core behavior of PredictionIO. It is very
>> likely
>> >>> that
>> >>> # you need to change these to fit your site.
>> >>>
>> >>> # SPARK_HOME: Apache Spark is a hard dependency and must be
>> configured.
>> >>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>> >>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>> >>>
>> >>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>> >>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>> >>>
>> >>> # ES_CONF_DIR: You must configure this if you have advanced
>> configuration
>> >>> for
>> >>> # your Elasticsearch setup.
>> >>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>> >>>
>> >>> # HADOOP_CONF_DIR: You must configure this if you intend to run
>> >>> PredictionIO
>> >>> # with Hadoop 2.
>> >>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>> >>>
>> >>> # HBASE_CONF_DIR: You must configure this if you intend to run
>> >>> PredictionIO
>> >>> # with HBase on a remote cluster.
>> >>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
>> >>>
>> >>> # Filesystem paths where PredictionIO uses as block storage.
>> >>> PIO_FS_BASEDIR=$HOME/.pio_store
>> >>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>> >>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>> >>>
>> >>> # PredictionIO Storage Configuration
>> >>> #
>> >>> # This section controls programs that make use of PredictionIO's
>> built-in
>> >>> # storage facilities. Default values are shown below.
>> >>> #
>> >>> # For more information on storage configuration please refer to
>> >>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>> >>>
>> >>> # Storage Repositories
>> >>>
>> >>> # Default is to use PostgreSQL
>> >>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>> >>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>> >>>
>> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>> >>>
>> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>> >>>
>> >>> # Storage Data Sources
>> >>>
>> >>> # PostgreSQL Default Settings
>> >>> # Please change "pio" to your database name in
>> >>> PIO_STORAGE_SOURCES_PGSQL_URL
>> >>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>> >>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>> >>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>> >>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>> >>>
>> >>> # MySQL Example
>> >>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>> >>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>> >>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>> >>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>> >>>
>> >>> # Elasticsearch Example
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>> >>> #
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
>> vendors/elasticsearch-5.5.2
>> >>> # Optional basic HTTP auth
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>> >>> # Elasticsearch 1.x Example
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>> >>>
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
>> vendors/elasticsearch-1.4.4
>> >>>
>> >>> # Local File System Example
>> >>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>> >>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>> >>>
>> >>> # HBase Example
>> >>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>> >>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
>> >>>
>> >>> # AWS S3 Example
>> >>> # PIO_STORAGE_SOURCES_S3_TYPE=s3
>> >>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>> >>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>> >>>
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Jim
>> >>>
>> >>>
>> >>> --
>> >>> Jim Miller
>> >>
>> >>
>> >>
>> >
>>
>>

Mime
View raw message