predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marius Rabenarivo <mariusrabenar...@gmail.com>
Subject Re: [PIO 0.10.0][Universtal Recommender 0.5] Exception when using remote Elasticsearch while training
Date Fri, 07 Apr 2017 05:57:25 GMT
I tried to run the command

curl -X GET http://engine-server:9200

and I'm getting a JSON response but when I'm running the command

pio train

I got

org.elasticsearch.hadoop.rest.EsHadoopTransportException:
java.net.NoRouteToHostException: No route to host (Host unreachable)

exception

2017-04-07 9:18 GMT+04:00 Marius Rabenarivo <mariusrabenarivo@gmail.com>:

> I tried using these configurations in the sparkConf section of engine.json
> It has the same effect as the code I mentioned below but I still getting
> the same exception.
>
> 2017-04-07 5:03 GMT+04:00 Pat Ferrel <pat@occamsmachete.com>:
>
>> BTW the docs for sparkConf are here: http://actionml.com/docs
>> /ur_config#ur-spark-conf
>>
>>
>> On Apr 6, 2017, at 4:52 PM, Pat Ferrel <pat@occamsmachete.com> wrote:
>>
>> PIO in the past used the “transport client” of Elasticsearch, which is on
>> port 9300 by default, The UR uses the REST client for some things like
>> rapid writes to ES using Spark using a lib supplied by Elasticsearch.
>>
>> So, Spark needs to have config that the Elasticsearch library can read.
>> This can be done most easily by putting it into a sparkConf section of
>> engine.json
>>
>> Like this:
>>
>>   "sparkConf": {
>>     "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
>>     "spark.kryo.registrator": "org.apache.mahout.sparkbindings.io
>> .MahoutKryoRegistrator",
>>     "spark.kryo.referenceTracking": "false",
>>     "spark.kryoserializer.buffer": "300m",
>>     "es.index.auto.create": “true”,
>>     “es.nodes”: “node1,node2”  <========== add this
>>   },
>>
>> The list is comma separated host ip addresses of DNS names so no spaces.
>> If you are using the default port 9200 you don’t need to specify it. These
>> values are defined by Elasticsearch so see their docs if you have any
>> question about what can be configured in this way. For instance you can
>> also specify password and username in this manner.
>>
>>
>> On Apr 6, 2017, at 4:27 PM, Marius Rabenarivo <mariusrabenarivo@gmail.com>
>> wrote:
>>
>> Hello,
>>
>> I try to run training from a remote machine.
>>
>> It fails in later stage due to this exception :
>>
>> org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection
>> error (check network and/or proxy settings)- all nodes failed; tried
>> [[engine-server:9300]]
>>
>> According to the stack trace the error is coming from this line :
>>
>> https://github.com/actionml/universal-recommender/blob/maste
>> r/src/main/scala/EsClient.scala#L179
>>
>> It was using a default configuration to access ES.
>>
>> I tried to replace this line with :
>>
>>
>>     val elasticSearchConf = Storage.getConfig("ELASTICSEARCH").get
>>
>>     val conf = Map(
>>       "es.mapping.id" -> "id",
>>       "es.nodes" -> elasticSearchConf.properties.get("HOSTS").head,
>>       "es.port" -> elasticSearchConf.properties.get("PORTS").head
>>     )
>>
>>     indexRDD.saveToEs(newIndexURI, conf)
>>
>> but I still getting the same error.
>>
>> Regards,
>>
>> Marius
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "actionml-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to actionml-user+unsubscribe@googlegroups.com.
>> To post to this group, send email to actionml-user@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/actionml-user/CAC-ATVES0FM4iFtXGoU9-URikEq%2B2d6NaV10Eh1
>> S4_t1Ub4PcA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/actionml-user/CAC-ATVES0FM4iFtXGoU9-URikEq%2B2d6NaV10Eh1S4_t1Ub4PcA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "actionml-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to actionml-user+unsubscribe@googlegroups.com.
>> To post to this group, send email to actionml-user@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/actionml-user/EA18C433-834A-4917-99C3-A2460C281667%40occ
>> amsmachete.com
>> <https://groups.google.com/d/msgid/actionml-user/EA18C433-834A-4917-99C3-A2460C281667%40occamsmachete.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>

Mime
View raw message