heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neng Lu <freen...@gmail.com>
Subject Re: Specifying Operator Resource in DSL
Date Thu, 21 Sep 2017 22:57:38 GMT
Just add some thoughts here: for ordinary heron topologies, the definition
of a heron job and the request of resources usage for each component are
separated: `TopologyBuilder` for job definition, `Config` for resource

In the dsl case, if we could also do something similar that separates the
dsl job creation and resources request, it would be really good. With this
separation, people has the flexibility of providing different configs for
the same job.

On Wed, Sep 20, 2017 at 1:48 PM, Sanjeev Kulkarni <sanjeevrk@gmail.com>

> Hi folks,
> One of the great features of the lower level spout/bolt interface in Heron
> is the ability to specify resources needed on a per component basis. This
> feature is very helpful for tuning large topologies and is heavily used
> inside Twitter.
> Currently the DSL does not have this flexibility. I wanted to get opinions
> about how we can add this.
> There are probably several ways to do it. I'm listing a few approaches that
> have come to my mind. Please feel free to add more.
> 1) Currently some of our operators are simple(like flatMap, map, filter
> operators), others are a little complicated(like transform where users can
> perform setup/cleanup). We can take the approach of adding the ability to
> specify resources only for complex operators. Thus transform could have two
> variants. The current one which just takes a transform function and another
> that takes in a resource parameter as well. The rest of other
> operators(map/flatmap/filter, etc) will remain the same. The advantage of
> this is that the interface explosion is minimal and controlled. The cons is
> that if you need to control the resources of a particular operator, you are
> forced to use transform.
> 2) Another approach would be to add a variant that takes in a Resource
> parameter to all operators. Pros is that this gives fine grained control to
> all operators. Cons is the interface blow up.
> Thoughts?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message