kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task
Date Sat, 05 Jan 2019 14:55:50 GMT
This is still open.

If you agree that we don't need it any longer, please update the wiki
pages accordingly and move the KIP into table "Discarded KIPs":
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-DiscardedKIPs


-Mattias

On 1/3/19 10:41 PM, Richard Yu wrote:
> Hi Matthias,
> 
> I guess this is no longer necessary. I am open to anything honestly.
> I suppose we should close it (if its not already).
> 
> 
> On Fri, Oct 19, 2018 at 11:06 AM Matthias J. Sax <matthias@confluent.io>
> wrote:
> 
>> Any thought on my last email about discarding this KIP?
>>
>>
>> -Matthias
>>
>> On 9/14/18 11:44 AM, Matthias J. Sax wrote:
>>> Hi,
>>>
>>> we recently had a discussion on a different ticket to reduce the size of
>>> the metadata we need to send:
>>> https://issues.apache.org/jira/browse/KAFKA-7149
>>>
>>> It seems, that we actually don't need to include the number of stores in
>>> the metadata, but that we can compute the number of stores locally on
>>> each instance.
>>>
>>> With this insight, we should still try to exploit this knowledge during
>>> task assignment, however, this would be an internal change that does not
>>> require a KIP. Thus, I think that we can discard this KIP.
>>>
>>> Thoughts?
>>>
>>>
>>> -Matthias
>>>
>>> On 6/10/18 5:20 PM, Matthias J. Sax wrote:
>>>> Richard,
>>>>
>>>> KIP-268 got merged and thus this KIP is unblocked.
>>>>
>>>> I just re-read it and think it needs some updates with regard to the
>>>> upgrade path (ie, you should mention why upgrading is covered).
>>>>
>>>> It would also be useful to discuss how the store information is used
>>>> during assignment. Atm, the KIP only discussed that the information
>>>> should be added, but this is only half of the story from my point of
>> view.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>> On 3/22/18 9:15 PM, Matthias J. Sax wrote:
>>>>> Hi Richard,
>>>>>
>>>>> with KIP-268 in place (should be accepted soon) the upgrade path is
>>>>> covered. Thus, you can update your KIP accordingly, referring to
>> KIP-268.
>>>>>
>>>>> Can you also update your KIP similar to KIP-268 to cover the old and
>> new
>>>>> metadata format?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> -Matthias
>>>>>
>>>>>
>>>>> On 2/24/18 4:07 PM, Richard Yu wrote:
>>>>>> I didn't really get what "upgrade strategy" was at the time that
>> Guozhang
>>>>>> mentioned it, so I wrote the above dialogue from my first
>> understanding. I
>>>>>> changed it to "upgrade strategy must be provided". Currently,
>> however, I do
>>>>>> not have anything in mind to facilitate upgrading older Kafka
>> brokers. If
>>>>>> you have anything in mind, please let me know.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax <
>> matthias@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks a lot for this KIP.
>>>>>>>
>>>>>>> I am not sure what you mean by
>>>>>>>
>>>>>>>> which could potentially break older versions of Kafka brokers
>>>>>>>
>>>>>>> The metadata that is exchange, is not interpreted by the brokers.
The
>>>>>>> problem with upgrading the metadata format affect only Kafka
Streams
>>>>>>> instances.
>>>>>>>
>>>>>>> If we don't provide an upgrade strategy, changing the metadata
format
>>>>>>> required to stop all running application instances, before the
>> instances
>>>>>>> can be restarted with the new code. However, this implies downtime
>> for
>>>>>>> an application and is thus not acceptable.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>> On 2/24/18 11:11 AM, Richard Yu wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I would like to discuss a KIP I've submitted :
>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>> 262%3A+Metadata+should+include+number+of+state+stores+for+task
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Richard Yu
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
> 


Mime
View raw message