openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominic Kim <>
Subject Re: Re: Backpressure for slow activation storage in Invoker
Date Tue, 25 Jun 2019 04:38:14 GMT
Let me share a few ideas on them.

Regarding option B1, I think it can scale out better than option B2.
If I understood correctly, scaling out of the service will be highly
dependent on Kafka.
Since the number of consumers is limited to the number of partitions, the
number of service nodes will be also limited to the number of partitions.

So in the case of B2, if we create a new topic with some partition numbers,
we cannot scale out the service nodes more than that.
At some point, we may need to alter the number of partitions and it's not
easy in Kafka.
(Since the activation processing here is asynchronous, we may bear some
downtime(1~2s) to alter the partition. Then it would be fine.)

In the case of B1, there will be many controller topics with their own
Since controllers can be scaled out, there will be more topics, and the
activation service can scale out accordingly.
But in this case, we need to manually control the topic assignment.
(Not partition assignment, it will be done by Kafka.)

Let's say we have 3 controller topics with 2 partitions each.
For HA, it would be great to have at least two nodes.
At first, both nodes will take care of all three topics.
Based on the partition assignment plan in Kafka, both nodes will fetch
activation messages without any duplication.
As controllers are scaled out, two nodes may not be enough to take care of
all topics.
At this point, we need to scale out the service nodes more.
Then we need to do logical partitioning for topics.

For example, the node1 and 2 will take care of topic0 ~ 1 and node3 and 4
will take care of topic2 ~ 3.
In this way, we can guarantee the minimum HA and scale out the nodes as
Among them, topic partitions will be also assigned by Kafka.

So in short,
For B1, we can scale out the service as controllers are scaled out, but it
would be much complex to manually assign topics.
And one node may have more than one Kafka consumers.

For B2, scaling might be limited unless we have a big enough number of
partitions at topic creation time.
But if we can bear some downtime, this might not be a problem and this
option will be a lot simpler.

Best regards

2019년 6월 24일 (월) 오후 6:50, Chetan Mehrotra <>님이

> Okie so we can then aim for adding an optional support for storing
> activations via a separate service.
> Currently we also send the activation result on respective controller
> topic. With this change we would also be sending same activation
> record on another topic. So we have another choice to make
> Option B1 - Activations via controller topic
> --------------------------------------------------------
> Here we avoid a new topic and instead have a service which listen to
> all controller topics for activation records. However that would be
> tricky to implement and also tricky to scale out. As scaling out such
> a service by running multiple copies would not be easy in terms of
> sharding/partitioning
> Here the benefit is that we reduce the duplicate writes on Kafka.
> Option B2 - Introduce a new topic altogether
> -----------------------------------------------------------
> We introduce a new topic to which all invokers write the activation
> records (like the case for user-events). Then implementing a new
> service to read from a single (possibly partitioned topic) would be
> easier.
> My suggestion is to go for B2 for now.
> Any feedback on that?
> Chetan Mehrotra
> On Fri, Jun 21, 2019 at 11:46 PM Rodric Rabbah <> wrote:
> >
> > > Can we handle these in same way as user events? Maybe exactly like user
> > events, as in use a single service to process both topics.
> >
> > good call - the user events already contains much of the activation
> record
> > (if not all modulo the logs)?
> >
> > -r

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message