spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject Re: Spark handling parallel requests
Date Sun, 18 Oct 2015 07:28:23 GMT
hi Akhlis 
its a must to push data to a socket as i am using php as a web service to push data to socket
, then spark catch the data on that socket and process it , is there a way to push data from
php to kafka directly ? --  Best Regards, -- Tarek Abouzeid 

     On Sunday, October 18, 2015 10:26 AM, "" <>

 hi Xiao,1- requests are not similar at all , but they use solr and do commit sometimes 2-
no caching is required3- the throughput must be very high yeah , the requests are tiny but
the system may receive 100 request/sec , does kafka support listening to a socket ? -- 
Best Regards, -- Tarek Abouzeid 

     On Monday, October 12, 2015 10:50 AM, Xiao Li <> wrote:

 Hi, Tarek, 
It is hard to answer your question. Are these requests similar? Caching your results or intermediate
results in your applications? Or does that mean your throughput requirement is very high?
Throttling the number of concurrent requests? ...
As Akhil said, Kafka might help in your case. Otherwise, you need to read the designs or even
source codes of Kafka and Spark Streaming. 
 Best wishes, 
Xiao Li

2015-10-11 23:19 GMT-07:00 Akhil Das <>:

Instead of pushing your requests to the socket, why don't you push them to a Kafka or any
other message queue and use spark streaming to process them?
ThanksBest Regards
On Mon, Oct 5, 2015 at 6:46 PM, <> wrote:

Hi ,
i am using Scala , doing a socket program to catch multiple requests at same time and then
call a function which uses spark to handle each process , i have a multi-threaded server to
handle the multiple requests and pass each to spark , but there's a bottleneck as the spark
doesn't initialize a sub task for the new request , is it even possible to do parallel processing
using single spark job ?Best Regards, --  Best Regards, -- Tarek Abouzeid


View raw message