spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sateesh Kavuri <>
Subject Re: Spark streaming multi-tasking during I/O
Date Sat, 22 Aug 2015 12:04:51 GMT
Thanks Akhil. Does this mean that the executor running in the VM can spawn
two concurrent jobs on the same core? If this is the case, this is what we
are looking for. Also, which version of Spark is this flag in?


On Sat, Aug 22, 2015 at 1:44 AM, Akhil Das <>

> You can look at the spark.streaming.concurrentJobs by default it runs a
> single job. If set it to 2 then it can run 2 jobs parallely. Its an
> experimental flag, but go ahead and give it a try.
> On Aug 21, 2015 3:36 AM, "Sateesh Kavuri" <>
> wrote:
>> Hi,
>> My scenario goes like this:
>> I have an algorithm running in Spark streaming mode on a 4 core virtual
>> machine. Majority of the time, the algorithm does disk I/O and database
>> I/O. Question is, during the I/O, where the CPU is not considerably loaded,
>> is it possible to run any other task/thread so as to efficiently utilize
>> the CPU?
>> Note that one DStream of the algorithm runs completely on a single CPU
>> Thank you,
>> Sateesh

View raw message