predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Banerjee <ashut...@juggernaut.in>
Subject ElasticSearch Nodes not Found - Universal Recommender Template
Date Tue, 17 Jan 2017 15:16:38 GMT
Hi,

We are facing an intermittent issue while using the universal recommender
template on our production servers. Every once in a while we get an error
that none of the ElasticSearch nodes are reachable
(org.elasticsearch.client.transport.NoNodeAvailableException: None of the
configured nodes are available: [])

Our elasticsearch cluster is hosted on a elasticsearch hosting
provider(Qbox) and at the time of receiving this error all the shards were
green and reachable from the server where predictio-io template is hosted.
There was no shortage of RAM or disk space on that server when this issue
occurred.
The issue gets resolved on performing the following steps:
1) pio undeploy
2) pio-stop-all
3) pio-start-all
4) pio deploy

Pio version used is 0.10.0-incubating and universal recommender version is
0.5.0
We did face this issue earlier as well and could not figure out why the ES
nodes became unreachable after a particular period.
The pio-env config for ES is below:

PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=<cluster-name>
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=<hosted-es-domain>
PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=<port> // Transport Layer Port Used
Here
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=<local-es-path>

We are currently using ElasticSearch version 1.5.2 on our hosted servers as
version 1.4.0 used by the pio is not supported on the hosted service. We
are using the Transport Layer port to communicate with the ES nodes as
opposed to calling the https endpoint.

Please help us figure out the underlying cause for this issue.

Thanks,
Ashutosh

Mime
View raw message