ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Gura <ag...@gridgain.com>
Subject Re: Using IgniteContext in... well, Java context
Date Fri, 09 Oct 2015 10:11:47 GMT

Ignite provides Java friendly RDD API. See JavaIgniteContext and
JavaIgniteRDD classes.

On Fri, Oct 9, 2015 at 12:26 AM, Konstantin Boudnik <cos@apache.org> wrote:

> Guys,
> I've tried to play a little bit with IgniteContext and the whole IgniteRDD
> stuff from the perspective of not touching Scala ever again. And here's
> what I
> have found: IgniteContext isn't usable from Java (or Groovy for that
> matter).
> And it isn't an attempt to critique Ignite's RDD implementation, because we
> have to follow the design patterns setup by the 3rd party platform, Spark
> in
> this case.
> If I want to submit a Java-based job into spark cluster I need to do
> something
> like this:
> import org.apache.ignite.spark.*
> import org.apache.spark.SparkConf
> import org.apache.spark.api.java.JavaSparkContext
> SparkConf sparkConf = new SparkConf().setAppName('SharedCache')
> sparkConf.setMaster('spark://master.spark.mydomain.com:7077')
> JavaSparkContext jsc = new JavaSparkContext(sparkConf)
> so far so good. Now, if I want to use Ignite in it, I should be doing
> def ic = new IgniteContext<T1, T2>(jsc, "spark-ignite-config.xml")
> and that's where I hit the wall, because JavaSparkContext isn't a subtype
> of
> SparkContext in Spark world, and I can not cast one to another.
> Well then, perhaps I can use SparkContext and hope for the best? Turns out
> I
> can not, because the use of the SparkContext gets me into the swamp of
> Scala
> type-system, with long-mnemonic names like Function1 and Tuple2 (I reckon
> the
> day will come soon, when they will have Function27 and so on). This leads
> me
> to believe the current implementation of IgniteRDD is only good to be used
> from Scala, unless I am completely wrong and don't know what I am talking
> about. Which might be quite possible, of course.
> My question is very simple: is there a way to use IgniteRDD from
> Java-language
> family, e.g. Java and Groovy, or there has to be JavaIgniteRDD
> implementation
> of it much like the state of things is in the Spark itself?
> Thanks for any feedback
> --
>   Cos

Andrey Gura
GridGain Systems, Inc.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message