predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Miller <jemiller1...@gmail.com>
Subject Re: Unable to connect to all storage backends successfully
Date Tue, 26 Sep 2017 23:42:01 GMT
Shinsuke, you rock!  That was it.

Thank you so much,

Jim

-- 
Jim Miller

On September 24, 2017 at 7:29:28 PM, Shinsuke Sugaya (shinsuke.sugaya@gmail.com) wrote:

Hi,  

> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300  

Elasticsearch 5 uses 9200 port.  
Could you try 9200 port?  

Thanks,  
shinsuke  


2017-09-24 3:02 GMT+09:00 Jim Miller <jemiller1963@gmail.com>:  
> Hi Donald,  
>  
> Tried just now and received the following error:  
>  
> vagrant:~/ $ pio status  
> [13:34:52]  
> [INFO] [Management$] Inspecting PredictionIO...  
> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at  
> /home/vagrant/pio/PredictionIO-0.12.0-incubating  
> [INFO] [Management$] Inspecting Apache Spark...  
> [INFO] [Management$] Apache Spark is installed at  
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.7  
> [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum requirement  
> of 1.3.0)  
> [INFO] [Management$] Inspecting storage backend connections...  
> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...  
> [ERROR] [Management$] Unable to connect to all storage backends  
> successfully.  
> The following shows the error message from the storage backend.  
>  
> error while performing request (java.lang.RuntimeException)  
>  
> Dumping configuration of initialized storage backend sources.  
> Please make sure they are correct.  
>  
> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->  
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.5.2,  
> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE -> 

> elastic search  
>  
>  
> HERE IS MY PIO-ENV.SH  
>  
> # PredictionIO Main Configuration  
> #  
> # This section controls core behavior of PredictionIO. It is very likely  
> that  
> # you need to change these to fit your site.  
>  
> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.  
> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7  
> SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7  
>  
> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar  
> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar  
>  
> # ES_CONF_DIR: You must configure this if you have advanced configuration  
> for  
> # your Elasticsearch setup.  
> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2  
> # HADOOP_CONF_DIR=/opt/hadoop  
>  
> # HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO  
> # with HBase on a remote cluster.  
> HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf  
>  
> # Filesystem paths where PredictionIO uses as block storage.  
> PIO_FS_BASEDIR=$HOME/.pio_store  
> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines  
> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp  
>  
> # PredictionIO Storage Configuration  
> #  
> # This section controls programs that make use of PredictionIO's built-in  
> # storage facilities. Default values are shown below.  
> #  
> # For more information on storage configuration please refer to  
> # http://predictionio.incubator.apache.org/system/anotherdatastore/  
>  
> # Storage Repositories  
>  
> # Default is to use PostgreSQL  
> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta  
> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH  
>  
> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event  
> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE  
>  
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model  
> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS  
>  
> # Storage Data Sources  
>  
> # PostgreSQL Default Settings  
> # Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL  
> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and  
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly  
> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc  
> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio  
> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio  
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio  
>  
> # MySQL Example  
> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc  
> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio  
> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio  
> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio  
>  
> # Elasticsearch Example  
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch  
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost  
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200  
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http  
> #  
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2  
> # Optional basic HTTP auth  
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name  
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret  
> # Elasticsearch 1.x Example  
> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch  
> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster  
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost  
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300  
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2  
>  
> # Local File System Example  
> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs  
> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models  
>  
> # HBase Example  
> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase  
> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1  
>  
> # AWS S3 Example  
> # PIO_STORAGE_SOURCES_S3_TYPE=s3  
> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket  
> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model  
>  
>  
> ELASTICSEARCH.YML  
>  
> #======================== Elasticsearch Configuration  
> =========================  
> #  
> # NOTE: Elasticsearch comes with reasonable defaults for most settings.  
> # Before you set out to tweak and tune the configuration, make sure  
> you  
> # understand what are you trying to accomplish and the consequences.  
> #  
> # The primary way of configuring a node is via this file. This template  
> lists  
> # the most important settings you may want to configure for a production  
> cluster.  
> #  
> # Please consult the documentation for further information on configuration  
> options:  
> # https://www.elastic.co/guide/en/elasticsearch/reference/index.html  
> #  
> # ---------------------------------- Cluster  
> -----------------------------------  
> #  
> # Use a descriptive name for your cluster:  
> #  
> cluster.name: firstCluster  
> #  
> # ------------------------------------ Node  
> ------------------------------------  
> #  
> # Use a descriptive name for the node:  
> #  
> #node.name: node-1  
> #  
> # Add custom attributes to the node:  
> #  
> #node.attr.rack: r1  
> #  
> # ----------------------------------- Paths  
> ------------------------------------  
> #  
> # Path to directory where to store the data (separate multiple locations by  
> comma):  
> #  
> #path.data: /path/to/data  
> #  
> # Path to log files:  
> #  
> #path.logs: /path/to/logs  
> #  
> # ----------------------------------- Memory  
> -----------------------------------  
> #  
> # Lock the memory on startup:  
> #  
> #bootstrap.memory_lock: true  
> #  
> # Make sure that the heap size is set to about half the memory available  
> # on the system and that the owner of the process is allowed to use this  
> # limit.  
> #  
> # Elasticsearch performs poorly when the system is swapping the memory.  
> #  
> # ---------------------------------- Network  
> -----------------------------------  
> #  
> # Set the bind address to a specific IP (IPv4 or IPv6):  
> #  
> #network.host: 192.168.0.1  
> #  
> # Set a custom port for HTTP:  
> #  
> #http.port: 9200  
> #  
> # For more information, consult the network module documentation.  
> #  
> # --------------------------------- Discovery  
> ----------------------------------  
> #  
> # Pass an initial list of hosts to perform discovery when new node is  
> started:  
> # The default list of hosts is ["127.0.0.1", "[::1]"]  
> #  
> #discovery.zen.ping.unicast.hosts: ["host1", "host2"]  
> #  
> # Prevent the "split brain" by configuring the majority of nodes (total  
> number of master-eligible nodes / 2 + 1):  
> #  
> #discovery.zen.minimum_master_nodes: 3  
> #  
> # For more information, consult the zen discovery module documentation.  
> #  
> # ---------------------------------- Gateway  
> -----------------------------------  
> #  
> # Block initial recovery after a full cluster restart until N nodes are  
> started:  
> #  
> #gateway.recover_after_nodes: 3  
> #  
> # For more information, consult the gateway module documentation.  
> #  
> # ---------------------------------- Various  
> -----------------------------------  
> #  
> # Require explicit names when deleting indices:  
> #  
> #action.destructive_requires_name: true  
>  
>  
> Thanks for all of your help!  
>  
> Jim  
> --  
> Jim Miller  
>  
> On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com)  
> wrote:  
>  
> Hi Donald,  
>  
> I did not. I will read the release notes and update accordingly.  
>  
> Thanks!  
>  
> Jim  
>  
> --  
> Jim Miller  
>  
> On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org) wrote:  
>  
> Hey Jim,  
>  
> Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in  
> 0.12 so the default build will use ES 5.x support.  
>  
> See the upcoming release notes:  
> https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes
 
>  
> Regards,  
> Donald  
>  
> On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <jemiller1963@gmail.com> wrote:  
>>  
>> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:  
>>  
>> https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
 
>>  
>> --  
>> Jim Miller  
>>  
>> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com)  
>> wrote:  
>>  
>> meaning is “firstcluster” the cluster name in your Elasticsearch  
>> configuration?  
>>  
>>  
>> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <vaghawan781@gmail.com> wrote: 

>>  
>> I think the problem is with Elasticsearch, are you sure the cluster exists  
>> in elasticsearch configuration?  
>>  
>> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <jemiller1963@gmail.com>  
>> wrote:  
>>>  
>>> Hi,  
>>>  
>>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:  
>>>  
>>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4  
>>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0  
>>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6  
>>>  
>>>  
>>> All starts with no errors but with pio status I get:  
>>>  
>>> [INFO] [Management$] Inspecting PredictionIO...  
>>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at  
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating  
>>> [INFO] [Management$] Inspecting Apache Spark...  
>>> [INFO] [Management$] Apache Spark is installed at  
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
 
>>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum  
>>> requirement of 1.3.0)  
>>> [INFO] [Management$] Inspecting storage backend connections...  
>>> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...  
>>> [ERROR] [Management$] Unable to connect to all storage backends  
>>> successfully.  
>>> The following shows the error message from the storage backend.  
>>>  
>>> Connection closed  
>>> (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)  
>>>  
>>> Dumping configuration of initialized storage backend sources.  
>>> Please make sure they are correct.  
>>>  
>>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->  
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4,
 
>>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE
->  
>>> elasticsearch  
>>>  
>>>  
>>> Can anyone give me an idea of what I need to fix this issue? Here is  
>>>  
>>>  
>>> # PredictionIO Main Configuration  
>>> #  
>>> # This section controls core behavior of PredictionIO. It is very likely  
>>> that  
>>> # you need to change these to fit your site.  
>>>  
>>> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.  
>>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7  
>>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6  
>>>  
>>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar  
>>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar  
>>>  
>>> # ES_CONF_DIR: You must configure this if you have advanced configuration  
>>> for  
>>> # your Elasticsearch setup.  
>>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf  
>>>  
>>> # HADOOP_CONF_DIR: You must configure this if you intend to run  
>>> PredictionIO  
>>> # with Hadoop 2.  
>>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf  
>>>  
>>> # HBASE_CONF_DIR: You must configure this if you intend to run  
>>> PredictionIO  
>>> # with HBase on a remote cluster.  
>>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf  
>>>  
>>> # Filesystem paths where PredictionIO uses as block storage.  
>>> PIO_FS_BASEDIR=$HOME/.pio_store  
>>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines  
>>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp  
>>>  
>>> # PredictionIO Storage Configuration  
>>> #  
>>> # This section controls programs that make use of PredictionIO's built-in  
>>> # storage facilities. Default values are shown below.  
>>> #  
>>> # For more information on storage configuration please refer to  
>>> # http://predictionio.incubator.apache.org/system/anotherdatastore/  
>>>  
>>> # Storage Repositories  
>>>  
>>> # Default is to use PostgreSQL  
>>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta  
>>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH  
>>>  
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event  
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE  
>>>  
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model  
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS  
>>>  
>>> # Storage Data Sources  
>>>  
>>> # PostgreSQL Default Settings  
>>> # Please change "pio" to your database name in  
>>> PIO_STORAGE_SOURCES_PGSQL_URL  
>>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and  
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly  
>>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc  
>>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio  
>>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio  
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio  
>>>  
>>> # MySQL Example  
>>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc  
>>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio  
>>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio  
>>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio  
>>>  
>>> # Elasticsearch Example  
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch  
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost  
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200  
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http  
>>> #  
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
 
>>> # Optional basic HTTP auth  
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name  
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret  
>>> # Elasticsearch 1.x Example  
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch  
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster  
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost  
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300  
>>>  
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
 
>>>  
>>> # Local File System Example  
>>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs  
>>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models  
>>>  
>>> # HBase Example  
>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase  
>>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0  
>>>  
>>> # AWS S3 Example  
>>> # PIO_STORAGE_SOURCES_S3_TYPE=s3  
>>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket  
>>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model  
>>>  
>>>  
>>> Thanks,  
>>>  
>>> Jim  
>>>  
>>>  
>>> --  
>>> Jim Miller  
>>  
>>  
>>  
>  

Mime
View raw message