ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pragmaticbigdata <amits...@gmail.com>
Subject Re: Apache Spark & Ignite Integration
Date Fri, 18 Nov 2016 13:56:02 GMT
a. No. I have not executed any tests. I am doing a theoretical study first to
understand the memory footprint and data movement between the spark & ignite
nodes

c. So basically there is no use case when working with spark (for data
processing) and ignite (for in-memory data storage) that can benefit from
ignite transactions. Since spark is non-transactional, I am trying to
understand how can use the ACID transactional support feature of ignite when
performing data processing in spark. Let me know if you think otherwise.


> To be honest, I'm not quite sure you should use IgniteRDD because it
> sounds like your use case is bigger than that (indexed SQL, transactions,
> etc.). Did you consider using pure Ignite API without integrating it with
> Spark? 

Yes, I have performed tests with pure Ignite API but the performance didn't
turn out to be well. Additionally we do have complex requirements in the
analytics space that could be served well with Spark. Are you suggesting
that IgniteRDD (in other words - spark integration) should be used when the
data set is small? From various blogs/articles I understand that the main
benefit of using spark with Ignite is to easily share data across spark jobs
where ignite is used for in-memory data storage and spark for its high speed
data processing capabilities.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Apache-Spark-Ignite-Integration-tp8556p9082.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message