predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <...@occamsmachete.com>
Subject Re: [PIO 0.10.0][Universtal Recommender 0.5] Exception when using remote Elasticsearch while training
Date Thu, 06 Apr 2017 23:52:31 GMT
PIO in the past used the “transport client” of Elasticsearch, which is on port 9300 by
default, The UR uses the REST client for some things like rapid writes to ES using Spark using
a lib supplied by Elasticsearch.

So, Spark needs to have config that the Elasticsearch library can read. This can be done most
easily by putting it into a sparkConf section of engine.json

Like this:

  "sparkConf": {
    "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
    "spark.kryo.registrator": "org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator",
    "spark.kryo.referenceTracking": "false",
    "spark.kryoserializer.buffer": "300m",
    "es.index.auto.create": “true”,
    “es.nodes”: “node1,node2”  <========== add this
  },
 
The list is comma separated host ip addresses of DNS names so no spaces. If you are using
the default port 9200 you don’t need to specify it. These values are defined by Elasticsearch
so see their docs if you have any question about what can be configured in this way. For instance
you can also specify password and username in this manner.


On Apr 6, 2017, at 4:27 PM, Marius Rabenarivo <mariusrabenarivo@gmail.com> wrote:

Hello,

I try to run training from a remote machine.

It fails in later stage due to this exception :

org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network
and/or proxy settings)- all nodes failed; tried [[engine-server:9300]]

According to the stack trace the error is coming from this line :

https://github.com/actionml/universal-recommender/blob/master/src/main/scala/EsClient.scala#L179
<https://github.com/actionml/universal-recommender/blob/master/src/main/scala/EsClient.scala#L179>

It was using a default configuration to access ES.

I tried to replace this line with :


    val elasticSearchConf = Storage.getConfig("ELASTICSEARCH").get

    val conf = Map(
      "es.mapping.id <http://es.mapping.id/>" -> "id",
      "es.nodes" -> elasticSearchConf.properties.get("HOSTS").head,
      "es.port" -> elasticSearchConf.properties.get("PORTS").head
    )

    indexRDD.saveToEs(newIndexURI, conf)

but I still getting the same error.

Regards,

Marius

-- 
You received this message because you are subscribed to the Google Groups "actionml-user"
group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-user+unsubscribe@googlegroups.com
<mailto:actionml-user+unsubscribe@googlegroups.com>.
To post to this group, send email to actionml-user@googlegroups.com <mailto:actionml-user@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/CAC-ATVES0FM4iFtXGoU9-URikEq%2B2d6NaV10Eh1S4_t1Ub4PcA%40mail.gmail.com
<https://groups.google.com/d/msgid/actionml-user/CAC-ATVES0FM4iFtXGoU9-URikEq%2B2d6NaV10Eh1S4_t1Ub4PcA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.


Mime
View raw message