ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From roshan joe <impdocs2...@gmail.com>
Subject implement shared RDD in spark streaming in scala
Date Wed, 01 Nov 2017 20:22:26 GMT
I am very new to Ignite and went through the documentation and samples on
the web. Below is the use case I am hoping to resolve using shared spark
RDDs.

   - Spark Streaming App-1 writes incremental records, which does not
   already exist (by using a hash value generated inside the App) to a shared
   RDD.


   - Spark Streaming App-2 looks up the data in this shared RDD using key
   columns and gets additional column values from the shared RDD populated by
   the App-1 above.


   - Spark Streaming App-3, Spark Streaming App-4 etc do the same lookup
   against shared RDD, same as the above.

I am sure this use-case has been implemented before and am trying to avoid
spending time to build this from scratch. Is there some sample code to do
this using scala that can be shared? Thank you very much!

Mime
View raw message