predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <...@occamsmachete.com>
Subject Re: [PIO 0.10.0][Universtal Recommender 0.5] Exception when using remote Elasticsearch while training
Date Mon, 10 Apr 2017 15:32:39 GMT
RE Elasticsearch, there is no need to have the clustername and hostname the same, in fact I
have never done this and we run larger clusters of ES. Are you using /etc/hosts or something
other than DNS to resolve the hostname or some other mechanism?

I guess if it’s working, it’s not an immediate issue but just wanted to make that clear.


On Apr 10, 2017, at 8:27 AM, Marius Rabenarivo <mariusrabenarivo@gmail.com> wrote:

Yes, I set the clustername in Elasticsearch config too.

Thank you for your advices about fields.

2017-04-10 19:07 GMT+04:00 Pat Ferrel <pat@occamsmachete.com <mailto:pat@occamsmachete.com>>:
Do you have the clustername set in Elasticsearch config also? It is changed in /usr/local/elasticsearch/config/elasticsearch.yml
Setting this in both places will decouple server name from cluster name. Unfortunately changing
a cluster name will call for a complete reset of PredictionIO data since it will loose track
of engine metadata. I have found it easiest to export/backup then delete all apps and re-install
or erase any table in hbase and elasticsearch. You have to get the clustername right before
you start because it’s a pain to change.

Also the fields must correspond to arrays of strings, not an array of true or false, an array
of one element, a string of “T” or “F” would work for this. For fields the property
must be attached to all items by using a $set event, then your visible and sponsored will
work more like what you expect. They are not working at all now.


On Apr 9, 2017, at 10:43 PM, Marius Rabenarivo <mariusrabenarivo@gmail.com <mailto:mariusrabenarivo@gmail.com>>
wrote:

Hello,

Find attached the pio-env.sh and engine.json

that we are using.

2017-04-08 18:46 GMT+04:00 Marius Rabenarivo <mariusrabenarivo@gmail.com <mailto:mariusrabenarivo@gmail.com>>:
I'll share them monday.

2017-04-07 22:23 GMT+04:00 Pat Ferrel <pat@occamsmachete.com <mailto:pat@occamsmachete.com>>:
We use a clustername different from hostname all the time. In fact if you have more than one
node you will have to make them different since there are multiple hostnames and one clustername.

The clustername should be something unique to your application because ES has the sometimes
helpful, but often maddeningly bad and hidden, habit of merging nodes into a cluster by clustername.
For instance the default for ES is to merge all nodes on a LAN into one cluster.

Anyway can you share your engine.json and the result of curl with exactly the same hostname?
You need to add the clustername to pio-env.sh too so can you share that as well?


On Apr 7, 2017, at 12:25 AM, Marius Rabenarivo <mariusrabenarivo@gmail.com <mailto:mariusrabenarivo@gmail.com>>
wrote:

It works when using the default port 9200 and by changing the Elasticsearch cluster name to
the same as the hostname as suggested here

http://stackoverflow.com/questions/25885411/elasticsearch-error-failed-to-connect-to-master-no-route-to-host
<http://stackoverflow.com/questions/25885411/elasticsearch-error-failed-to-connect-to-master-no-route-to-host>

2017-04-07 9:57 GMT+04:00 Marius Rabenarivo <mariusrabenarivo@gmail.com <mailto:mariusrabenarivo@gmail.com>>:
I tried to run the command

curl -X GET http://engine-server:9200 <http://engine-server:9200/>

and I'm getting a JSON response but when I'm running the command

pio train

I got 

org.elasticsearch.hadoop.rest.EsHadoopTransportException: java.net <http://java.net/>.NoRouteToHostException:
No route to host (Host unreachable)

exception

2017-04-07 9:18 GMT+04:00 Marius Rabenarivo <mariusrabenarivo@gmail.com <mailto:mariusrabenarivo@gmail.com>>:
I tried using these configurations in the sparkConf section of engine.json
It has the same effect as the code I mentioned below but I still getting the same exception.

2017-04-07 5:03 GMT+04:00 Pat Ferrel <pat@occamsmachete.com <mailto:pat@occamsmachete.com>>:
BTW the docs for sparkConf are here: http://actionml.com/docs/ur_config#ur-spark-conf <http://actionml.com/docs/ur_config#ur-spark-conf>


On Apr 6, 2017, at 4:52 PM, Pat Ferrel <pat@occamsmachete.com <mailto:pat@occamsmachete.com>>
wrote:

PIO in the past used the “transport client” of Elasticsearch, which is on port 9300 by
default, The UR uses the REST client for some things like rapid writes to ES using Spark using
a lib supplied by Elasticsearch.

So, Spark needs to have config that the Elasticsearch library can read. This can be done most
easily by putting it into a sparkConf section of engine.json

Like this:

  "sparkConf": {
    "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
    "spark.kryo.registrator": "org.apache.mahout.sparkbindings.io <http://sparkbindings.io/>.MahoutKryoRegistrator",
    "spark.kryo.referenceTracking": "false",
    "spark.kryoserializer.buffer": "300m",
    "es.index.auto.create": “true”,
    “es.nodes”: “node1,node2”  <========== add this
  },
 
The list is comma separated host ip addresses of DNS names so no spaces. If you are using
the default port 9200 you don’t need to specify it. These values are defined by Elasticsearch
so see their docs if you have any question about what can be configured in this way. For instance
you can also specify password and username in this manner.


On Apr 6, 2017, at 4:27 PM, Marius Rabenarivo <mariusrabenarivo@gmail.com <mailto:mariusrabenarivo@gmail.com>>
wrote:

Hello,

I try to run training from a remote machine.

It fails in later stage due to this exception :

org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network
and/or proxy settings)- all nodes failed; tried [[engine-server:9300]]

According to the stack trace the error is coming from this line :

https://github.com/actionml/universal-recommender/blob/master/src/main/scala/EsClient.scala#L179
<https://github.com/actionml/universal-recommender/blob/master/src/main/scala/EsClient.scala#L179>

It was using a default configuration to access ES.

I tried to replace this line with :


    val elasticSearchConf = Storage.getConfig("ELASTICSEARCH").get

    val conf = Map(
      "es.mapping.id <http://es.mapping.id/>" -> "id",
      "es.nodes" -> elasticSearchConf.properties.get("HOSTS").head,
      "es.port" -> elasticSearchConf.properties.get("PORTS").head
    )

    indexRDD.saveToEs(newIndexURI, conf)

but I still getting the same error.

Regards,

Marius

-- 
You received this message because you are subscribed to the Google Groups "actionml-user"
group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-user+unsubscribe@googlegroups.com
<mailto:actionml-user+unsubscribe@googlegroups.com>.
To post to this group, send email to actionml-user@googlegroups.com <mailto:actionml-user@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/CAC-ATVES0FM4iFtXGoU9-URikEq%2B2d6NaV10Eh1S4_t1Ub4PcA%40mail.gmail.com
<https://groups.google.com/d/msgid/actionml-user/CAC-ATVES0FM4iFtXGoU9-URikEq%2B2d6NaV10Eh1S4_t1Ub4PcA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.


-- 
You received this message because you are subscribed to the Google Groups "actionml-user"
group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-user+unsubscribe@googlegroups.com
<mailto:actionml-user+unsubscribe@googlegroups.com>.
To post to this group, send email to actionml-user@googlegroups.com <mailto:actionml-user@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/EA18C433-834A-4917-99C3-A2460C281667%40occamsmachete.com
<https://groups.google.com/d/msgid/actionml-user/EA18C433-834A-4917-99C3-A2460C281667%40occamsmachete.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.





-- 
You received this message because you are subscribed to the Google Groups "actionml-user"
group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-user+unsubscribe@googlegroups.com
<mailto:actionml-user+unsubscribe@googlegroups.com>.
To post to this group, send email to actionml-user@googlegroups.com <mailto:actionml-user@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/CAC-ATVGHNYHJ-V78B1knbpqWGn_cpkvYCC_%2BW3Tmdw6kywcSTw%40mail.gmail.com
<https://groups.google.com/d/msgid/actionml-user/CAC-ATVGHNYHJ-V78B1knbpqWGn_cpkvYCC_%2BW3Tmdw6kywcSTw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.




-- 
You received this message because you are subscribed to the Google Groups "actionml-user"
group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-user+unsubscribe@googlegroups.com
<mailto:actionml-user+unsubscribe@googlegroups.com>.
To post to this group, send email to actionml-user@googlegroups.com <mailto:actionml-user@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/CAC-ATVG8madmx19TqijGra4n6TxDSnxvziB4RNggnvAosB7Etg%40mail.gmail.com
<https://groups.google.com/d/msgid/actionml-user/CAC-ATVG8madmx19TqijGra4n6TxDSnxvziB4RNggnvAosB7Etg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
<engine.json><pio-env.sh>



-- 
You received this message because you are subscribed to the Google Groups "actionml-user"
group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionml-user+unsubscribe@googlegroups.com
<mailto:actionml-user+unsubscribe@googlegroups.com>.
To post to this group, send email to actionml-user@googlegroups.com <mailto:actionml-user@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/actionml-user/CAC-ATVESU16R_w_Hs%2B49a7unJmpUdjghASDjXRiYhdZRrEAggQ%40mail.gmail.com
<https://groups.google.com/d/msgid/actionml-user/CAC-ATVESU16R_w_Hs%2B49a7unJmpUdjghASDjXRiYhdZRrEAggQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.


Mime
View raw message