incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Vavassori <>
Subject Information about Apache S4 and Helix
Date Wed, 11 Dec 2013 16:50:54 GMT
> It's my understanding that a cluster is a group of nodes and each node has
>> the same application-code copy; this means that if I want to partition the
>> ProcessingElements between nodes I need to group them in different
>> clusters.
>> So, mapping S4 elements into a "classical" Stream Processing naming
>> (Nodes,
>> Operators, Slides...), would be having one application (Operator) per
>> cluster and configure the ProcessingElements as singleton (1 Slide per
>> Node).
> you should use a key to partition your stream. Use the KeyFinder to
> identify keys in events.

but this is Stream Partitioning, not Processing Element Partitioning. I
could use this system, but for my needs I should then multiply each message
for the number of partitions and I still wouldn’t be able to control PE
allocation between types in the same nodes.

E.g. if I have:

* 2 PE types: PE1 and PE2, configured as singletons.
* 3 nodes

in each node both PE types can be allocated, so 6 instead of 3. Is there a
way to allocate only one PE per node?

Sergio Vavassori

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message