spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jay vyas <>
Subject Re: Does Spark Streaming calculate during a batch?
Date Thu, 13 Nov 2014 15:55:30 GMT
1) Your have a receiver thread. That thread might use alot of CPU, or not,
depending on how  you implement the thread in onStart.

2) Every 5 minutes, spark will submit a job which process
every RDD which was created (i.e using the store() call) in the
receiver .  That job will run asynchronously to the receiver, which
is still working to produce new RDDs for the next batch,

So, maybe you're monitoring the CPU only on the
spark workers which is running the batch jobs, and not
on the spark worker which is doing the RDD ingestion?

On Thu, Nov 13, 2014 at 10:35 AM, Michael Campbell <> wrote:

> I was running a proof of concept for my company with spark streaming, and
> the conclusion I came to is that spark collects data for the
> batch-duration, THEN starts the data-pipeline calculations.
> My batch size was 5 minutes, and the CPU was all but dead for 5, then when
> the 5 minutes were up the CPU's would spike for a while presumably doing
> the calculations.
> Is this presumption true, or is it running the data through the
> calculation pipeline before the batch is up?
> What could lead to the periodic CPU spike - I had a reduceByKey, so was it
> doing that only after all the batch data was in?
> Thanks

jay vyas

View raw message