heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjeev Kulkarni <sanjee...@gmail.com>
Subject Specifying Operator Resource in DSL
Date Wed, 20 Sep 2017 20:48:32 GMT
Hi folks,
One of the great features of the lower level spout/bolt interface in Heron
is the ability to specify resources needed on a per component basis. This
feature is very helpful for tuning large topologies and is heavily used
inside Twitter.
Currently the DSL does not have this flexibility. I wanted to get opinions
about how we can add this.
There are probably several ways to do it. I'm listing a few approaches that
have come to my mind. Please feel free to add more.
1) Currently some of our operators are simple(like flatMap, map, filter
operators), others are a little complicated(like transform where users can
perform setup/cleanup). We can take the approach of adding the ability to
specify resources only for complex operators. Thus transform could have two
variants. The current one which just takes a transform function and another
that takes in a resource parameter as well. The rest of other
operators(map/flatmap/filter, etc) will remain the same. The advantage of
this is that the interface explosion is minimal and controlled. The cons is
that if you need to control the resources of a particular operator, you are
forced to use transform.
2) Another approach would be to add a variant that takes in a Resource
parameter to all operators. Pros is that this gives fine grained control to
all operators. Cons is the interface blow up.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message