kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Tan <c...@confluent.io>
Subject Re: [DISCUSS] KIP-580: Exponential Backoff for Kafka Clients
Date Mon, 23 Mar 2020 18:38:18 GMT
+1 (non-binding)

> On Mar 19, 2020, at 7:27 PM, Sanjana Kaundinya <skaundinya@gmail.com> wrote:
> 
> Ah yes that makes sense. I’ll update the KIP to reflect this.
> 
> Thanks,
> Sanjana
> 
> On Thu, Mar 19, 2020 at 5:48 PM Guozhang Wang <wangguoz@gmail.com> wrote:
> 
>> Following the formula you have in the KIP, if it is simply:
>> 
>> MIN(retry.backoff.max.ms, (retry.backoff.ms * 2**(failures - 1)) * random(
>> 0.8, 1.2))
>> 
>> then the behavior would stay consistent at retry.backoff.max.ms.
>> 
>> 
>> Guozhang
>> 
>> On Thu, Mar 19, 2020 at 5:46 PM Sanjana Kaundinya <skaundinya@gmail.com>
>> wrote:
>> 
>>> If that’s the case then what should we base the starting point as?
>>> Currently in the KIP the starting point is retry.backoff.ms and it
>>> exponentially goes up to retry.backoff.max.ms. If retry.backoff.max.ms
>> is
>>> smaller than retry.backoff.ms then that could pose a bit of a problem
>>> there right?
>>> 
>>> On Mar 19, 2020, 5:44 PM -0700, Guozhang Wang <wangguoz@gmail.com>,
>> wrote:
>>>> Thanks Sanjana, I did not capture the part that Jason referred to, so
>>>> that's my bad :P
>>>> 
>>>> Regarding your last statement, I actually feel that instead of take the
>>>> larger of the two, we should respect "retry.backoff.max.ms" even if it
>>> is
>>>> smaller than "retry.backoff.ms". I do not have a very strong rationale
>>>> except it is logically more aligned to the config names.
>>>> 
>>>> 
>>>> Guozhang
>>>> 
>>>> 
>>>> On Thu, Mar 19, 2020 at 5:39 PM Sanjana Kaundinya <
>> skaundinya@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hey Jason and Guozhang,
>>>>> 
>>>>> Jason is right, I took this inspiration from KIP-144 (
>>>>> 
>>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-144%3A+Exponential+backoff+for+broker+reconnect+attempts
>>>>> )
>>>>> which had the same logic in order to preserve the existing behavior.
>> In
>>>>> this case however, if we are thinking to completely eliminate the
>>> static
>>>>> backoff behavior, we can do that and as Jason mentioned put it in the
>>>>> release notes and not add any special logic. In addition I agree that
>>> we
>>>>> should take the larger of the two of `retry.backoff.ms` and `
>>>>> retry.backoff.max.ms`. I'll update the KIP to reflect this and make
>> it
>>>>> clear that the old static retry backoff is getting replaced by the
>> new
>>>>> dynamic retry backoff.
>>>>> 
>>>>> Thanks,
>>>>> Sanjana
>>>>> On Thu, Mar 19, 2020 at 4:23 PM Jason Gustafson <jason@confluent.io>
>>>>> wrote:
>>>>> 
>>>>>> Hey Guozhang,
>>>>>> 
>>>>>> I was referring to this:
>>>>>> 
>>>>>>> For users who have not set retry.backoff.ms explicitly, the
>>> default
>>>>>> behavior will change so that the backoff will grow up to 1000 ms.
>> For
>>>>> users
>>>>>> who have set retry.backoff.ms explicitly, the behavior will remain
>>> the
>>>>>> same
>>>>>> as they could have specific requirements.
>>>>>> 
>>>>>> I took this to mean that for users who have overridden `
>>> retry.backoff.ms
>>>>> `
>>>>>> to 50ms (say), we will change the default `retry.backoff.max.ms`
>> to
>>> 50ms
>>>>>> as
>>>>>> well in order to preserve existing backoff behavior. Is that not
>>> right?
>>>>> In
>>>>>> any case, I agree that we can use the maximum of the two values as
>>> the
>>>>>> effective `retry.backoff.max.ms` to handle the case when the
>>> configured
>>>>>> value of `retry.backoff.ms` is larger than the default of 1s.
>>>>>> 
>>>>>> -Jason
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Thu, Mar 19, 2020 at 3:29 PM Guozhang Wang <wangguoz@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>>> Hey Jason,
>>>>>>> 
>>>>>>> My understanding is a bit different here: even if user has an
>>> explicit
>>>>>>> overridden "retry.backoff.ms", the exponential mechanism still
>>>>> triggers
>>>>>>> and
>>>>>>> the backoff would be increased till "retry.backoff.max.ms"; and
>>> if the
>>>>>>> specified "retry.backoff.ms" is already larger than the "
>>>>>>> retry.backoff.max.ms", we would still take "retry.backoff.max.ms
>> ".
>>>>>>> 
>>>>>>> So if the user does override the "retry.backoff.ms" to a value
>>> larger
>>>>>> than
>>>>>>> 1s and is not aware of the new config, she would be surprised
to
>>> see
>>>>> the
>>>>>>> specified value seemingly not being respected, but she could
>> still
>>>>> learn
>>>>>>> that afterwards by reading the release notes introducing this
KIP
>>>>>> anyways.
>>>>>>> 
>>>>>>> 
>>>>>>> Guozhang
>>>>>>> 
>>>>>>> On Thu, Mar 19, 2020 at 3:10 PM Jason Gustafson <
>>> jason@confluent.io>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi Sanjana,
>>>>>>>> 
>>>>>>>> The KIP looks good to me. I had just one question about the
>>> default
>>>>>>>> behavior. As I understand, if the user has specified `
>>>>> retry.backoff.ms
>>>>>> `
>>>>>>>> explicitly, then we will not apply the default max backoff.
As
>>> such,
>>>>>>>> there's no way to get the benefit of this feature if you
are
>>>>> providing
>>>>>> a
>>>>>>> `
>>>>>>>> retry.backoff.ms` unless you also provide `
>> retry.backoff.max.ms
>>> `.
>>>>> That
>>>>>>>> makes sense if you assume the user is unaware of the new
>>>>> configuration,
>>>>>>> but
>>>>>>>> it is surprising otherwise. Since it's not a semantic change
>> and
>>>>> since
>>>>>>> the
>>>>>>>> default you're proposing of 1s is fairly low already, I wonder
>> if
>>>>> it's
>>>>>>> good
>>>>>>>> enough to mention the new configuration in the release notes
>> and
>>> not
>>>>>> add
>>>>>>>> any special logic. What do you think?
>>>>>>>> 
>>>>>>>> -Jason
>>>>>>>> 
>>>>>>>> On Thu, Mar 19, 2020 at 1:56 PM Sanjana Kaundinya <
>>>>>> skaundinya@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Thank you for the comments Guozhang.
>>>>>>>>> 
>>>>>>>>> I’ll leave this KIP out for discussion till the end
of the
>>> week and
>>>>>>> then
>>>>>>>>> start a vote for this early next week.
>>>>>>>>> 
>>>>>>>>> Sanjana
>>>>>>>>> 
>>>>>>>>> On Mar 18, 2020, 3:38 PM -0700, Guozhang Wang <
>>> wangguoz@gmail.com
>>>>>> ,
>>>>>>>> wrote:
>>>>>>>>>> Hello Sanjana,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the proposed KIP, I think that makes a
lot of
>>> sense --
>>>>>> as
>>>>>>>> you
>>>>>>>>>> mentioned in the motivation, we've indeed seen many
issues
>>> with
>>>>>>> regard
>>>>>>>> to
>>>>>>>>>> the frequent retries, with bounded exponential backoff
in
>> the
>>>>>>> scenario
>>>>>>>>>> where there's a long connectivity issue we would
>> effectively
>>>>> reduce
>>>>>>> the
>>>>>>>>>> request load by 10 given the default configs.
>>>>>>>>>> 
>>>>>>>>>> For higher-level Streams client and Connect frameworks,
>>> today we
>>>>>> also
>>>>>>>>> have
>>>>>>>>>> a retry logic but that's used in a slightly different
way.
>>> For
>>>>>>> example
>>>>>>>> in
>>>>>>>>>> Streams, we tend to handle the retry logic at the
>>> thread-level
>>>>> and
>>>>>>>> hence
>>>>>>>>>> very likely we'd like to change that mechanism in
KIP-572
>>>>> anyways.
>>>>>>> For
>>>>>>>>>> producer / consumer / admin clients, I think just
applying
>>> this
>>>>>>>>> behavioral
>>>>>>>>>> change across these clients makes lot of sense. So
I think
>>> can
>>>>> just
>>>>>>>> leave
>>>>>>>>>> the Streams / Connect out of the scope of this KIP
to be
>>>>> addressed
>>>>>> in
>>>>>>>>>> separate discussions.
>>>>>>>>>> 
>>>>>>>>>> I do not have further comments about this KIP :)
LGTM.
>>>>>>>>>> 
>>>>>>>>>> Guozhang
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Wed, Mar 18, 2020 at 12:09 AM Sanjana Kaundinya
<
>>>>>>>> skaundinya@gmail.com
>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Thanks for the feedback Boyang.
>>>>>>>>>>> 
>>>>>>>>>>> If there’s anyone else who has feedback regarding
this
>> KIP,
>>>>> would
>>>>>>>>> really
>>>>>>>>>>> appreciate it hearing it!
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Sanjana
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Mar 17, 2020 at 11:38 PM Boyang Chen
<
>>>>>> bchen11@outlook.com>
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Sounds great!
>>>>>>>>>>>> 
>>>>>>>>>>>> Get Outlook for iOS<https://aka.ms/o0ukef>
>>>>>>>>>>>> ________________________________
>>>>>>>>>>>> From: Sanjana Kaundinya <skaundinya@gmail.com>
>>>>>>>>>>>> Sent: Tuesday, March 17, 2020 5:54:35 PM
>>>>>>>>>>>> To: dev@kafka.apache.org <dev@kafka.apache.org>
>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-580: Exponential
Backoff for
>>> Kafka
>>>>>>>> Clients
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the explanation Boyang. One of
the most
>> common
>>>>>>> problems
>>>>>>>>> that
>>>>>>>>>>> we
>>>>>>>>>>>> have in Kafka is with respect to metadata
fetches. For
>>>>> example,
>>>>>>> if
>>>>>>>>> there
>>>>>>>>>>> is
>>>>>>>>>>>> a broker failure, all clients start to fetch
metadata
>> at
>>> the
>>>>>> same
>>>>>>>>> time
>>>>>>>>>>> and
>>>>>>>>>>>> it often takes a while for the metadata to
converge.
>> In a
>>>>> high
>>>>>>> load
>>>>>>>>>>>> cluster, there are also issues where the
volume of
>>> metadata
>>>>> has
>>>>>>>> made
>>>>>>>>>>>> convergence of metadata slower.
>>>>>>>>>>>> 
>>>>>>>>>>>> For this case, exponential backoff helps
as it reduces
>>> the
>>>>>> retry
>>>>>>>>> rate and
>>>>>>>>>>>> spaces out how often clients will retry,
thereby
>> bringing
>>>>> down
>>>>>>> the
>>>>>>>>> time
>>>>>>>>>>> for
>>>>>>>>>>>> convergence. Something that Jason mentioned
that would
>>> be a
>>>>>> great
>>>>>>>>>>> addition
>>>>>>>>>>>> here would be if the backoff should be “jittered”
as it
>>> was
>>>>> in
>>>>>>>>> KIP-144
>>>>>>>>>>> with
>>>>>>>>>>>> respect to exponential reconnect backoff.
This would
>> help
>>>>>> prevent
>>>>>>>> the
>>>>>>>>>>>> clients from being synchronized on when they
retry,
>>> thereby
>>>>>>> spacing
>>>>>>>>> out
>>>>>>>>>>> the
>>>>>>>>>>>> number of requests being sent to the broker
at the same
>>> time.
>>>>>>>>>>>> 
>>>>>>>>>>>> I’ll add this example to the KIP and flush
out more of
>>> the
>>>>>>> details
>>>>>>>> -
>>>>>>>>> so
>>>>>>>>>>>> it’s more clear.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mar 17, 2020, 1:24 PM -0700, Boyang Chen
<
>>>>>>>>> reluctanthero104@gmail.com
>>>>>>>>>>>> ,
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Thanks for the reply Sanjana. I guess
I would like to
>>>>>> rephrase
>>>>>>> my
>>>>>>>>>>>> question
>>>>>>>>>>>>> 2 and 3 as my previous response is a
little bit
>>>>> unactionable.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> My specific point is that exponential
backoff is not
>> a
>>>>> silver
>>>>>>>>> bullet
>>>>>>>>>>> and
>>>>>>>>>>>> we
>>>>>>>>>>>>> should consider using it to solve known
problems,
>>> instead
>>>>> of
>>>>>>>>> making the
>>>>>>>>>>>>> holistic changes to all clients in Kafka
ecosystem. I
>>> do
>>>>> like
>>>>>>> the
>>>>>>>>>>>>> exponential backoff idea and believe
this would be of
>>> great
>>>>>>>> value,
>>>>>>>>> but
>>>>>>>>>>>>> maybe we should focus on proposing some
existing
>>> modules
>>>>> that
>>>>>>> are
>>>>>>>>>>>> suffering
>>>>>>>>>>>>> from static retry, and only change them
in this first
>>> KIP.
>>>>> If
>>>>>>> in
>>>>>>>>> the
>>>>>>>>>>>>> future, some other component users believe
they are
>>> also
>>>>>>>>> suffering, we
>>>>>>>>>>>>> could get more minor KIPs to change the
behavior as
>>> well.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Boyang
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sun, Mar 15, 2020 at 12:07 AM Sanjana
Kaundinya <
>>>>>>>>>>> skaundinya@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the feedback Boyang, I
will revise the
>> KIP
>>>>> with
>>>>>>> the
>>>>>>>>>>>>>> mathematical relations as per your
suggestion. To
>>> address
>>>>>>> your
>>>>>>>>>>>> feedback:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1. Currently, with the default of
100 ms per retry
>>>>> backoff,
>>>>>>> in
>>>>>>>> 1
>>>>>>>>>>> second
>>>>>>>>>>>>>> we would have 10 retries. In the
case of using an
>>>>>> exponential
>>>>>>>>>>> backoff,
>>>>>>>>>>>> we
>>>>>>>>>>>>>> would have a total of 4 retries in
1 second. Thus
>> we
>>> have
>>>>>>> less
>>>>>>>>> than
>>>>>>>>>>>> half of
>>>>>>>>>>>>>> the amount of retries in the same
timeframe and can
>>>>> lessen
>>>>>>>> broker
>>>>>>>>>>>> pressure.
>>>>>>>>>>>>>> This calculation is done as following
(using the
>>> formula
>>>>>> laid
>>>>>>>>> out in
>>>>>>>>>>>> the
>>>>>>>>>>>>>> KIP:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Try 1 at time 0 ms, failures = 0,
next retry in 100
>>> ms
>>>>>>> (default
>>>>>>>>> retry
>>>>>>>>>>>> ms
>>>>>>>>>>>>>> is initially 100 ms)
>>>>>>>>>>>>>> Try 2 at time 100 ms, failures =
1, next retry in
>>> 200 ms
>>>>>>>>>>>>>> Try 3 at time 300 ms, failures =
2, next retry in
>>> 400 ms
>>>>>>>>>>>>>> Try 4 at time 700 ms, failures =
3, next retry in
>>> 800 ms
>>>>>>>>>>>>>> Try 5 at time 1500 ms, failures =
4, next retry in
>>> 1000
>>>>> ms
>>>>>>>>> (default
>>>>>>>>>>> max
>>>>>>>>>>>>>> retry ms is 1000 ms)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> For 2 and 3, could you elaborate
more about what
>> you
>>> mean
>>>>>>> with
>>>>>>>>>>> respect
>>>>>>>>>>>> to
>>>>>>>>>>>>>> client timeouts? I’m not very familiar
with the
>>> Streams
>>>>>>>>> framework, so
>>>>>>>>>>>> would
>>>>>>>>>>>>>> love to get more insight to how that
currently
>> works,
>>>>> with
>>>>>>>>> respect to
>>>>>>>>>>>>>> producer transactions, so I can appropriately
>> update
>>> the
>>>>>> KIP
>>>>>>> to
>>>>>>>>>>> address
>>>>>>>>>>>>>> these scenarios.
>>>>>>>>>>>>>> On Mar 13, 2020, 7:15 PM -0700, Boyang
Chen <
>>>>>>>>>>>> reluctanthero104@gmail.com>,
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Thanks for the KIP Sanjana. I
think the
>> motivation
>>> is
>>>>>> good,
>>>>>>>> but
>>>>>>>>>>> lack
>>>>>>>>>>>> of
>>>>>>>>>>>>>>> more quantitative analysis. For
instance:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1. How much retries we are saving
by applying the
>>>>>>> exponential
>>>>>>>>> retry
>>>>>>>>>>>> vs
>>>>>>>>>>>>>>> static retry? There should be
some mathematical
>>>>> relations
>>>>>>>>> between
>>>>>>>>>>> the
>>>>>>>>>>>>>>> static retry ms, the initial
exponential retry
>> ms,
>>> the
>>>>>> max
>>>>>>>>>>>> exponential
>>>>>>>>>>>>>>> retry ms in a given time interval.
>>>>>>>>>>>>>>> 2. How does this affect the client
timeout? With
>>>>>>> exponential
>>>>>>>>> retry,
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> client shall be getting easier
to timeout on a
>>> parent
>>>>>> level
>>>>>>>>> caller,
>>>>>>>>>>>> for
>>>>>>>>>>>>>>> instance stream attempts to retry
initializing
>>> producer
>>>>>>>>>>> transactions
>>>>>>>>>>>> with
>>>>>>>>>>>>>>> given 5 minute interval. With
exponential retry
>>> this
>>>>>>>> mechanism
>>>>>>>>>>> could
>>>>>>>>>>>>>>> experience more frequent timeout
which we should
>> be
>>>>>> careful
>>>>>>>>> with.
>>>>>>>>>>>>>>> 3. With regards to #2, we should
have more
>> detailed
>>>>>>> checklist
>>>>>>>>> of
>>>>>>>>>>> all
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> existing static retry scenarios,
and adjust the
>>> initial
>>>>>>>>> exponential
>>>>>>>>>>>> retry
>>>>>>>>>>>>>>> ms to make sure we won't get
easily timeout in
>> high
>>>>> level
>>>>>>> due
>>>>>>>>> to
>>>>>>>>>>> too
>>>>>>>>>>>> few
>>>>>>>>>>>>>>> attempts.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Boyang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Mar 13, 2020 at 4:38
PM Sanjana
>> Kaundinya <
>>>>>>>>>>>> skaundinya@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I’ve written a KIP about
introducing
>> exponential
>>>>>> backoff
>>>>>>>> for
>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>> clients. Would appreciate
any feedback on this.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-580%3A+Exponential+Backoff+for+Kafka+Clients
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Sanjana
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> -- Guozhang
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> -- Guozhang
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> -- Guozhang
>>> 
>> 
>> 
>> --
>> -- Guozhang
>> 


Mime
View raw message