incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Information about Apache S4 and Helix
Date Thu, 12 Dec 2013 17:54:23 GMT
There is a discussion about stream partitioning v/s PE partitioning in that
JIRA. I had some implementation that allows both types of partitioning
schemes.

https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=blob;f=subprojects/s4-tools/src/main/java/org/apache/s4/tools/helix/CreateTask.java;h=36a45bc5bef5e4c4f56bfb4c740abb315197ca21;hb=59bcb94cc727cb1600074e6da81330953aeaa566

When you add the task, you can use pe type s the name of the task and
specify the streams it is interested in.

Give it a try :-)



On Wed, Dec 11, 2013 at 8:50 AM, Sergio Vavassori <svavassori@conwet.com>wrote:

> >
> > It's my understanding that a cluster is a group of nodes and each node
> has
> >> the same application-code copy; this means that if I want to partition
> the
> >> ProcessingElements between nodes I need to group them in different
> >> clusters.
> >> So, mapping S4 elements into a "classical" Stream Processing naming
> >> (Nodes,
> >> Operators, Slides...), would be having one application (Operator) per
> >> cluster and configure the ProcessingElements as singleton (1 Slide per
> >> Node).
> >>
> >
> > you should use a key to partition your stream. Use the KeyFinder to
> > identify keys in events.
>
>
> but this is Stream Partitioning, not Processing Element Partitioning. I
> could use this system, but for my needs I should then multiply each message
> for the number of partitions and I still wouldn’t be able to control PE
> allocation between types in the same nodes.
>
> E.g. if I have:
>
> * 2 PE types: PE1 and PE2, configured as singletons.
> * 3 nodes
>
> in each node both PE types can be allocated, so 6 instead of 3. Is there a
> way to allocate only one PE per node?
>
> Regards,
> Sergio Vavassori
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message