spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feynman Liang <fli...@databricks.com>
Subject Re: Few basic spark questions
Date Mon, 13 Jul 2015 19:07:37 GMT
Hi Oded,

I'm not sure I completely understand your question, but it sounds like you
could have the READER receiver produce a DStream which is
windowed/processed in Spark Streaming and forEachRDD to do the OUTPUT.
However, streaming in SparkR is not currently supported (SPARK-6803
<https://issues.apache.org/jira/browse/SPARK-6803>) so I'm not too sure how
ANALYZER would fit in.

Feynman

On Sun, Jul 12, 2015 at 11:23 PM, Oded Maimon <oded@scene53.com> wrote:

> any help / idea will be appreciated :)
> thanks
>
>
> Regards,
> Oded Maimon
> Scene53.
>
> On Sun, Jul 12, 2015 at 4:49 PM, Oded Maimon <oded@scene53.com> wrote:
>
>> Hi All,
>> we are evaluating spark for real-time analytic. what we are trying to do
>> is the following:
>>
>>    - READER APP- use custom receiver to get data from rabbitmq (written
>>    in scala)
>>    - ANALYZER APP - use spark R application to read the data (windowed),
>>    analyze it every minute and save the results inside spark
>>    - OUTPUT APP - user spark application (scala/java/python) to read the
>>    results from R every X minutes and send the data to few external systems
>>
>> basically at the end i would like to have the READER COMPONENT as an app
>> that always consumes the data and keeps it in spark,
>> have as many ANALYZER COMPONENTS as my data scientists wants, and have
>> one OUTPUT APP that will read the ANALYZER results and send it to any
>> relevant system.
>>
>> what is the right way to do it?
>>
>> Thanks,
>> Oded.
>>
>>
>>
>>
>
> *This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they are
> addressed. Please note that any disclosure, copying or distribution of the
> content of this information is strictly forbidden. If you have received
> this email message in error, please destroy it immediately and notify its
> sender.*
>

Mime
View raw message