ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From luismattor <luismat...@gmail.com>
Subject Re: Use apache Ignite outside spark-shell
Date Thu, 26 May 2016 14:10:51 GMT

The problem is in IgniteContext.scala:

// Start ignite server node on each worker in server mode.
sparkContext.parallelize(1 to workers, workers).foreachPartition(it ⇒

As far as I understand, this line works as far as each executor holds a
single partition. That is, if executor owns more than one partition
IgniteContext will try to launch several Ignite nodes (in the same
process); if there are executors with no partitions, no Ignite instance
will be launched.

For example, Image a two worker cluster (node1, node2), each worker having
two cores. The idea is to launch an Ignite node on each executor. So we
sparkContext.parallelize(1 to 2, 2).foreachPartition(it ⇒ ignite())

It might be the chance the two partitions are handled by node1. Or it might
be the case node1 handle one partition, and node2 the other. (this is what
we want). So the question is how to make sure the latter scenario happens?


On 26 May 2016 at 05:19, vkulichenko [via Apache Ignite Users] <
ml-node+s70518n5228h30@n6.nabble.com> wrote:

> Hi,
> Why do you think this is happening? Do you have any exception? Generally,
> IgniteContext always reuses the Ignite client if it already exists in the
> current process.
> Note that you can still have several clients in several processes. Usually
> you will have a client on the driver and a client per each executor (not
> worker!).
> -Val
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5228.html
> To unsubscribe from Use apache Ignite outside spark-shell, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5208&code=bHVpc21hdHRvckBnbWFpbC5jb218NTIwOHwyNjY2MDQ0MTc=>
> .
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Use-apache-Ignite-outside-spark-shell-tp5208p5238.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
View raw message