incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <>
Subject Re: S4 newbie, some questions on scalability, db access..
Date Thu, 13 Oct 2011 10:57:50 GMT
(forwarded from the old google groups S4 project list, to the apache user

Hi Sundar,

>> I am new to S4, went through the S4 documentation recently. I have couple
>> of questions.. pardon me if my questions are dump..
>> 1. Whether a PN shall be created/added/deleted at run time?
by PN, if you mean Processing Node: you usually use scripts to start them.
In addition, there is a warm failover recovery scheme : a standby node is
incorporated to the cluster if an active node dies, and takes over the
workload of the defunct node.

2. I may need to access some data from DB when PE is processes the events.
>> Will that be a good approach or always use memory?
one of the driving design idea of S4 is to minimize latency by using local
memory in processing nodes and avoid disk I/O bottleneck. That also applies
to other streaming platforms. Therefore, it is usually not recommended to
integrate DB access in the event processing flow. But it is not impossible,
depending on you internal SLAs, and you could try to minimize induced
latency through prefetching / caching strategies.

3. Just to understand the scalability of S4, if it need to process feeds
>> from millions of subscribers, whether millions of PEs are required?
>> it depends on how you model your application. If you decide to key some
PEs by subscriber ID, then you may have millions of PEs, and that's not an
issue: S4 was designed for that. You will need to provision you S4 cluster

> 4) Looks like developer need to do all the processing logic (in
>> processEvent method), S4 just facilitates a platform to do it?
processing logic is indeed implemented in processEvent methods, but you can
also use some preexisting PEs for some standard operations on streams such
as JoinPE, ReroutePE etc...

Hope this helps!


View raw message