kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task
Date Fri, 19 Oct 2018 18:06:28 GMT
Any thought on my last email about discarding this KIP?


-Matthias

On 9/14/18 11:44 AM, Matthias J. Sax wrote:
> Hi,
> 
> we recently had a discussion on a different ticket to reduce the size of
> the metadata we need to send:
> https://issues.apache.org/jira/browse/KAFKA-7149
> 
> It seems, that we actually don't need to include the number of stores in
> the metadata, but that we can compute the number of stores locally on
> each instance.
> 
> With this insight, we should still try to exploit this knowledge during
> task assignment, however, this would be an internal change that does not
> require a KIP. Thus, I think that we can discard this KIP.
> 
> Thoughts?
> 
> 
> -Matthias
> 
> On 6/10/18 5:20 PM, Matthias J. Sax wrote:
>> Richard,
>>
>> KIP-268 got merged and thus this KIP is unblocked.
>>
>> I just re-read it and think it needs some updates with regard to the
>> upgrade path (ie, you should mention why upgrading is covered).
>>
>> It would also be useful to discuss how the store information is used
>> during assignment. Atm, the KIP only discussed that the information
>> should be added, but this is only half of the story from my point of view.
>>
>>
>> -Matthias
>>
>> On 3/22/18 9:15 PM, Matthias J. Sax wrote:
>>> Hi Richard,
>>>
>>> with KIP-268 in place (should be accepted soon) the upgrade path is
>>> covered. Thus, you can update your KIP accordingly, referring to KIP-268.
>>>
>>> Can you also update your KIP similar to KIP-268 to cover the old and new
>>> metadata format?
>>>
>>> Thanks!
>>>
>>> -Matthias
>>>
>>>
>>> On 2/24/18 4:07 PM, Richard Yu wrote:
>>>> I didn't really get what "upgrade strategy" was at the time that Guozhang
>>>> mentioned it, so I wrote the above dialogue from my first understanding.
I
>>>> changed it to "upgrade strategy must be provided". Currently, however, I
do
>>>> not have anything in mind to facilitate upgrading older Kafka brokers. If
>>>> you have anything in mind, please let me know.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax <matthias@confluent.io>
>>>> wrote:
>>>>
>>>>> Thanks a lot for this KIP.
>>>>>
>>>>> I am not sure what you mean by
>>>>>
>>>>>> which could potentially break older versions of Kafka brokers
>>>>>
>>>>> The metadata that is exchange, is not interpreted by the brokers. The
>>>>> problem with upgrading the metadata format affect only Kafka Streams
>>>>> instances.
>>>>>
>>>>> If we don't provide an upgrade strategy, changing the metadata format
>>>>> required to stop all running application instances, before the instances
>>>>> can be restarted with the new code. However, this implies downtime for
>>>>> an application and is thus not acceptable.
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>>
>>>>> On 2/24/18 11:11 AM, Richard Yu wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I would like to discuss a KIP I've submitted :
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>> 262%3A+Metadata+should+include+number+of+state+stores+for+task
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Richard Yu
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
> 


Mime
View raw message