Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
MIME-Version: 1.0
From: Valentin Kulichenko <valentin.kulichenko@gmail.com>
Date: Thu, 30 Nov 2017 15:07:43 -0800
Message-ID: <CABuYRcoGMj89dDt5pfnxNvT5UGyHA0beZv+EKZy0KrjhuX3VAg@mail.gmail.com>
Subject: Deprecate IgniteRDD in embedded mode
To: dev@ignite.apache.org
Content-Type: multipart/alternative; boundary="94eb2c0d00d0a09026055f3b5309"
archived-at: Thu, 30 Nov 2017 23:08:19 -0000

--94eb2c0d00d0a09026055f3b5309
Content-Type: text/plain; charset="UTF-8"

Igniters,

Currently we claim to support IgniteRDD in two modes: standalone and
embedded. Standalone means there is a separately running Ignite cluster,
and Spark start client node(s) to interact with it. In embedded node
everything runs within Spark, including Ignite server nodes that are
started embedded into Spark executors.

The latter case doesn't really work, mainly because the lifecycle of Spark
executors is not very predictable - Spark can start and stop them while
application is running. In case Ignite cluster is used to store data (which
is usually the case), this causes unnecessary rebalancing or even
unexpected data loss.

I propose to deprecate and eventually discontinue the embedded mode.
Luckily, standalone mode is the default one, so we can simply print out a
clear warning if one switches to embedded mode, and also mention this in
the docs.

Thoughts? If there are no objections, I will create a ticket and make the
change.

-Val

--94eb2c0d00d0a09026055f3b5309--