ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devl Devel <devl.developm...@gmail.com>
Subject Apache Ignite Integration with Spark RDDs
Date Sat, 11 Apr 2015 11:00:41 GMT
Hi Ingiters

Spark has an RDD interface which lets you plug in various back-ends like
HDFS and JDBC etc. I think it would be a great idea if the community could
leverage Ignites in-memory file system to provide blazing fast analytics
using both ingite and spark.

Here is the interface :
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala

The idea would be to compute splits and partitions from ignitefs and use
these in an RDD, an example is:

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala

Would like to know whether this is feasible, as it would be a useful
addition to both communities and help complement other implementations for
tachyon for example.

Cheers
Devl

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message