openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominic Kim <style9...@gmail.com>
Subject Re: Autonomous Container Scheduler v2 proposal
Date Thu, 29 Nov 2018 03:11:14 GMT
Dear Rodric.

We`ve just started benchmarking of the scheduler with a variety of
scenarios.
I will come with more results.

If you ask that because of scalability of Kafka, I have considered Kafka as
a very scalable solution.

Linkedin announced that they are utilizing a huge Kafka cluster.
https://events.static.linuxfound.org/sites/events/files/slides/Kafka%20At%20Scale.pdf

As per the document, they are utilizing 1100+ Kafka brokers, over 31,000
topics, 350,000+ partitions.

And as per my understanding, only a limited number of brokers participate
in message delivery at a time for a topic.
Since partitions are the unit of parallelism, if we have 5 partitions in a
topic, only maximum 5 brokers can deliver the message at the same time.
(Actually, consumers fetch the messages instead brokers deliver them, but
anyway.)

Since it's not required for all brokers to participate in all message
processing, when we need to scale out Kafka, we can just add more brokers.

For example, if we have 5 topics with 5 partitions each in 5 brokers.
There will be 1 partition for each topic in each broker. As more and more
topics are created, there will be more partitions in one broker.
Let's say the maximum number of partitions that one broker properly handles
in time is 10, then if there are 10 topics, message processing will start
becoming slow.
Then we can just add more brokers. If we add 1 broker, it can handle 10
partitions and it is equal to 2 topics in this example.
So whenever 2 new topics are created we can add a broker so that we can
keep the number of partitions that one broker holds same.
If we can keep the number of partitions in one broker at the same level, we
can guarantee the performance of Kafka as one broker will only participate
in the limited number of partition processing.

In this scene, metadata sharing for partitions might be considered as a
bottleneck.
But once partitions are assigned to brokers, it does not change over
time(if we don't explicitly rebalance them).
Then consumers can just statically read the location of partitions and
fetch data from target brokers.

Partitions are evenly distributed among brokers and each consumer will
access to a few of them only at a time, we can achieve massive scalability
with Kafka.

Since this is based on my understanding about Kafka rather than real
experience, if I observe any issues during benchmarking I will share them
as well.



Thanks
Regards
Dominic






2018년 11월 29일 (목) 오전 9:54, Rodric Rabbah <rodric@gmail.com>님이 작성:

> Thanks for sharing Dominic. I'll need to read this slowly to understand it.
> It does look like a bit of the proposal you've discussed in the past with 1
> topic per action name (per namespace). Have you seen this scale in your
> production to thousands of topics?
>
> -r
>
> On Wed, Nov 28, 2018 at 5:59 AM Dominic Kim <style9595@gmail.com> wrote:
>
> > Dear whiskers.
> >
> > I`ve just submitted the proposal for the next version of Autonomous
> > Container Scheduler.
> >
> >
> https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduler+v2
> >
> > We(Naver), already implemented the scheduler based on that proposal and
> > it's almost finished.
> > It's not all-round scheduler but we could observe many performance
> benefits
> > in a few cases.
> >
> > We(Naver) are willing to contribute it and preparing for the
> contribution.
> > I hope the proposal would help you guys understand the implementation.
> >
> > To contribute the scheduler, it is required to contribute SPI for
> > ContainerPool and ContainerProxy first.
> > I will start working on this.
> > And I will describe the details about each component in each subpage.
> > Currently, they are blank.
> >
> > Please be noted again that the scheduler is not for all cases.
> > But it would be useful in many cases until new implementation based on
> the
> > future architecture of OW would be landed.
> >
> >
> https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture?src=contextnavpagetreemode
> >
> > Any comments and feedback would be very grateful.
> >
> > Thanks
> > Regards
> > Dominic
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message