spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feynman Liang <>
Subject Re: Few basic spark questions
Date Mon, 13 Jul 2015 19:07:37 GMT
Hi Oded,

I'm not sure I completely understand your question, but it sounds like you
could have the READER receiver produce a DStream which is
windowed/processed in Spark Streaming and forEachRDD to do the OUTPUT.
However, streaming in SparkR is not currently supported (SPARK-6803
<>) so I'm not too sure how
ANALYZER would fit in.


On Sun, Jul 12, 2015 at 11:23 PM, Oded Maimon <> wrote:

> any help / idea will be appreciated :)
> thanks
> Regards,
> Oded Maimon
> Scene53.
> On Sun, Jul 12, 2015 at 4:49 PM, Oded Maimon <> wrote:
>> Hi All,
>> we are evaluating spark for real-time analytic. what we are trying to do
>> is the following:
>>    - READER APP- use custom receiver to get data from rabbitmq (written
>>    in scala)
>>    - ANALYZER APP - use spark R application to read the data (windowed),
>>    analyze it every minute and save the results inside spark
>>    - OUTPUT APP - user spark application (scala/java/python) to read the
>>    results from R every X minutes and send the data to few external systems
>> basically at the end i would like to have the READER COMPONENT as an app
>> that always consumes the data and keeps it in spark,
>> have as many ANALYZER COMPONENTS as my data scientists wants, and have
>> one OUTPUT APP that will read the ANALYZER results and send it to any
>> relevant system.
>> what is the right way to do it?
>> Thanks,
>> Oded.
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they are
> addressed. Please note that any disclosure, copying or distribution of the
> content of this information is strictly forbidden. If you have received
> this email message in error, please destroy it immediately and notify its
> sender.*

View raw message