Return-Path: X-Original-To: apmail-kafka-dev-archive@www.apache.org Delivered-To: apmail-kafka-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8A7C518A96 for ; Fri, 8 Jan 2016 22:11:28 +0000 (UTC) Received: (qmail 52078 invoked by uid 500); 8 Jan 2016 22:11:28 -0000 Delivered-To: apmail-kafka-dev-archive@kafka.apache.org Received: (qmail 51982 invoked by uid 500); 8 Jan 2016 22:11:28 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 51969 invoked by uid 99); 8 Jan 2016 22:11:27 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jan 2016 22:11:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 3E492C1437 for ; Fri, 8 Jan 2016 22:11:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=confluent-io.20150623.gappssmtp.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id bBAwo9XAZQua for ; Fri, 8 Jan 2016 22:11:17 +0000 (UTC) Received: from mail-oi0-f52.google.com (mail-oi0-f52.google.com [209.85.218.52]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 01BAB204D0 for ; Fri, 8 Jan 2016 22:11:16 +0000 (UTC) Received: by mail-oi0-f52.google.com with SMTP id k206so19650998oia.1 for ; Fri, 08 Jan 2016 14:11:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=confluent-io.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=d0NvLwJdtvv7I11+/7cDp8n8VLxNtHIN1eDLb5x1Ewc=; b=dYiIFWQSkR9A3VVsUVZ7PrjILQx5mf1ux+Mgdu0MMSXo3k3hzvChSg3NahNCjD6Q5R vg20YPFVx80Gr5wwHiYBN5ElBGfCg4PACbPJTR8AnzZnDk/0uCv24m+SDdGwJcLB2sA6 Df1Xz7ewffXJYtcHYFaDCHv1OtJNm97OSg4Z+pcSpnnIXTZgs9ImL4zyubwsS45b/z7d +DTJ9LX1Sq+X2dU7/G6Lz05v6GzvD1Vl6EX5R1pulQtSclj0poCxllkk9XXnF/mQTYbt ReTWC5csy6C1w/x7AZFTwG1zPeOy9vAERaks6Z0zHpuzaQcS/GWEZYBLvElfloZqd9Zy klTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=d0NvLwJdtvv7I11+/7cDp8n8VLxNtHIN1eDLb5x1Ewc=; b=TbqmoyIgmTZQBmVPSxRezt17idwX5z78IhiSTzB5LLjl9Thy8k6L+6tTzSLPVeecQB S7SPtA4CtLMzTKOX12wp4Dk6N5T9TmHNl2a9vz9IcM0+FXWFUFBnizS+M1Ts9PjIYaXS eMFq4RESnZ+J5Y65k0yRxyjREjaQi+7zAgb3arbajlXhFOSBrXq0fvTygHdwBmX8n1B1 zm+6fF2vT1ei3Sj5uT08ATFeWjMG0sY2q1f0Q/1GJUSilUwNSRHoWy+anzZkx04qsAh9 pRj6eNsuYdWu9K9LWzIn2FswlGxLBLbh95hcSDkQTeZtEBDNccLJuSItJpeXnVe0tDQ5 Ut1w== X-Gm-Message-State: ALoCoQmUva6tbFPSoNylodqMIl2APCBws94+v4+l6yr8nObtXedw1B/F0xWwVBYz0/eQnED42JA5Yo+wLfkfNkN9Dbzt7a5bYw== MIME-Version: 1.0 X-Received: by 10.202.85.146 with SMTP id j140mr76157794oib.4.1452291075990; Fri, 08 Jan 2016 14:11:15 -0800 (PST) Received: by 10.182.138.72 with HTTP; Fri, 8 Jan 2016 14:11:15 -0800 (PST) In-Reply-To: References: <62B2BD55-CB01-46C6-8CF6-44F104A8A782@gmail.com> <4F24B6BE-161C-4D85-AF8F-C0A41B273941@gmail.com> Date: Fri, 8 Jan 2016 14:11:15 -0800 Message-ID: Subject: Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message. From: Gwen Shapira To: dev@kafka.apache.org Content-Type: multipart/alternative; boundary=001a113d216eb20cde0528d9dd06 --001a113d216eb20cde0528d9dd06 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sounds good to me too. Seems pretty easy to add and can be useful for producers. On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy wrote: > Hi Anna, > > That sounds good to me - Becket/others any thoughts? > > Thanks, > > Joel > > On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner wrote: > > > Hi Becket and everyone, > > > > Could we please add the following functionality to this KIP. I think it > > would be very useful for the broker to return the timestamp in the ack = to > > the producer (in response: timestamp per partition) and propagate it ba= ck > > to client in RecordMetadata. This way, if timestamp type is > LogAppendTime, > > the producer client will see what timestamp was actually set -- and it > > would match the timestamp that consumer sees. Also, returning the > timestamp > > in RecordMetadata is also useful for timestamp type =3D CreateTime, sin= ce > > timestamp could be also set in KafkaProducer (if client set timestamp i= n > > ProducerRecord to 0). > > > > Since this requires protocol change as well, it will be better to > implement > > this as part of KIP-32, rather than proposing a new KIP. > > > > Thanks, > > Anna > > > > > > On Fri, Jan 8, 2016 at 12:53 PM, Joel Koshy wrote= : > > > > > +1 from me > > > > > > Looking through this thread it seems there was some confusion on the > > > migration discussion. This discussion in fact happened in the KIP-31 > > > discuss thread, not so much in the KIP hangout. There is considerable > > > overlap in discussions between KIP-3[1,2,3] so it makes sense to > > > cross-reference all of these. > > > > > > I'm finding the Apache list archive a little cumbersome to use (e.g., > the > > > current link in KIP-31 points to the beginning of September archives) > but > > > the emails discussing migration were in October: > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread > > > > > > Markmail has a better interface but interestingly it has not indexed > any > > of > > > the emails from August, September and early October ( > > > > > > > > > http://markmail.org/search/?q=3Dlist%3Aorg.apache.incubator.kafka-dev+dat= e%3A201509-201511+order%3Adate-backward > > > ). > > > Perhaps KIPs should include a permalink to the first message of the > > DISCUSS > > > thread. E.g., > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm= 5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E > > > > > > Also, just to clarify Jay's comments on the content of KIPs: I think > > having > > > a pseudo-code spec/implementation guide is useful (especially for > > > client-side KIPs). While the motivation should definitely capture =E2= =80=9Cwhy > we > > > are doing the KIP=E2=80=9D it probably shouldn=E2=80=99t have to exha= ustively capture > > =E2=80=9Cwhy > > > we are doing the KIP *this way*=E2=80=9D. i.e., some of the discussio= ns are > > > extremely nuanced and in this case spans multiple KIPs so links to > other > > > KIPs and the discuss threads and KIP hangout recordings are perhaps > > > sufficient to fill this gap - or maybe a new section that summarizes > the > > > discussions. > > > > > > Thanks, > > > > > > Joel > > > > > > On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao wrote: > > > > > > > Hi, Jiangjie, > > > > > > > > 52. Replacing MessageSet with o.a.k.common.record will be ideal. > > > > Unfortunately, we use MessageSet in SimpleConsumer, which is part o= f > > the > > > > public api. Replacing MessageSet with o.a.k.common.record will be a= n > > > > incompatible api change. So, we probably should do this after we > > > deprecate > > > > SimpleConsumer. > > > > > > > > My original question is actually whether we just bump up magic byte > in > > > > Message once to incorporate both the offset and the timestamp chang= e. > > It > > > > seems that the answer is yes. Could you reflect that in the KIP? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin > > wrote: > > > > > > > > > Thanks a lot for the careful reading, Jun. > > > > > Please see inline replies. > > > > > > > > > > > > > > > > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > > > > > > > > > > > Jiangjie, > > > > > > > > > > > > Thanks for the updated KIP. Overall, a +1 on the proposal. A fe= w > > > minor > > > > > > comments on the KIP. > > > > > > > > > > > > KIP-32: > > > > > > 50. 6.c says "The log rolling has to depend on the earliest > > > timestamp", > > > > > > which is inconsistent with KIP-33. > > > > > Corrected. > > > > > > > > > > > > 51. 8.b "If the time difference threshold is set to 0. The > > timestamp > > > in > > > > > the > > > > > > message is equivalent to LogAppendTime." If the time difference > is > > 0 > > > > and > > > > > > CreateTime is used, all messages will likely be rejected in thi= s > > > > > proposal. > > > > > > So, it's not equivalent to LogAppendTime. > > > > > Corrected. > > > > > > > > > > > > 52. Could you include the new value of magic byte in message > format > > > > > change? > > > > > > Also, do we have a single new message format that includes both > the > > > > > offset > > > > > > change (relative offset for inner messages) and the addition of > > > > > timestamp? > > > > > I am actually thinking about this when I am writing the patch. > > > > > The timestamp will be added to the o.a.k.common.record.Record and > > > > > Kafka.message.Message. The offset change is in > > > > > o.a.k.common.record.MemoryRecords and Kafka.message.MessageSet. T= o > > > avoid > > > > > unnecessary changes, my current patch did not merge them together > but > > > > > simply make sure the version of Record(Message) and > > > > > MemoryRecords(MessageSet) matches. > > > > > > > > > > Currently new clients uses classes in o.a.k.common.record, and th= e > > > broker > > > > > and old clients uses classes in kafka.message. > > > > > I am thinking about doing the followings: > > > > > 1. Migrate broker to use o.a.k.common.record. > > > > > 2. Add message format V0 and V1 to o.a.k.common.protocol.Protocol= s. > > > > > Ideally we should be able to define all the wire protocols in > > > > > o.a.k.common.protocol.Protocol. So instead of having Record class > to > > > > parse > > > > > byte arrays by itself, we can use Schema to parse the records. > > > > > > > > > > Would that be better? > > > > > > > > > > > > 53. Could you document the changes in ProducerRequest V2 and > > > > FetchRequest > > > > > > V2 (and the responses)? > > > > > Done. > > > > > > > > > > > > 54. In migration phase 1, step 2, does internal ApiVersion mean > > > > > > inter.broker.protocol.version? > > > > > Yes. > > > > > > > > > > > > 55. In canary step 2.b, it says "It will only see > > > > > > ProduceRequest/FetchRequest V1 from other brokers and clietns."= . > > But > > > in > > > > > > phase 2, a broker will receive FetchRequest V2 from other > brokers. > > > > > I meant when we canary a broker in phase 2, there will be only on= e > > > broker > > > > > entering phase 2, the other brokers will remain at phase 1. > > > > > > > > > > > > > > > > > > KIP-33: > > > > > > 60. The KIP still says maintaining index at "at minute > granularity" > > > > even > > > > > > though the index interval is configurable now. > > > > > Corrected. > > > > > > > > > > > > 61. In this design, it's possible for a log segment to have an > > empty > > > > time > > > > > > index. In the worse case, we may have to scan more than the > active > > > > > segment > > > > > > to recover the latest timestamp. > > > > > Corrected. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Mon, Jan 4, 2016 at 11:37 AM, Aditya Auradkar < > > > > > > aauradkar@linkedin.com.invalid> wrote: > > > > > > > > > > > >> Hey Becket/Anna - > > > > > >> > > > > > >> I have a few comments about the KIP. > > > > > >> > > > > > >> 1. (Minor) Can we rename the KIP? It's currently "Add CreateTi= me > > and > > > > > >> LogAppendTime etc..". This is actually the title of the now > > rejected > > > > > Option > > > > > >> 1. > > > > > >> 2. (Minor) Can we rename the proposed option? It isn't really > > > "option > > > > 4" > > > > > >> anymore. > > > > > >> 3. I'm not clear on what exactly happens to compressed message= s > > > > > >> when message.timestamp.type=3DLogAppendTime? Does every batch = get > > > > > >> recompressed because the inner message gets rewritten with the > > > server > > > > > >> timestamp? Or does the message set on disk have the timestamp > set > > to > > > > > -1. In > > > > > >> that case, what do we use as timestamp for the message? > > > > > >> 4. Do message.timestamp.type and max.message.time.difference.m= s > > > need > > > > > to be > > > > > >> per-topic configs? It seems that this is really a client confi= g > > > i.e. a > > > > > >> client is the source of timestamps not a topic. It could also > be a > > > > > >> broker-level config to keep things simple. > > > > > >> 5. The "Proposed Changes" section in the KIP tries to build a > > > > time-based > > > > > >> index for query but that is a separate proposal (KIP-33). Can = we > > > more > > > > > >> crisply identify what exactly will change when this KIP (and 3= 1) > > is > > > > > >> implemented? It isn't super clear to me at this point. > > > > > >> > > > > > >> Aside from that, I think the "Rejected Alternatives" section o= f > > the > > > > KIP > > > > > is > > > > > >> excellent. Very good insight into what options were discussed > and > > > > > rejected. > > > > > >> > > > > > >> Aditya > > > > > >> > > > > > >>> On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin < > > becket.qin@gmail.com> > > > > > wrote: > > > > > >>> > > > > > >>> Thanks Guozhang, Gwen and Neha for the comments. Sorry for la= te > > > reply > > > > > >>> because I only have occasional gmail access from my phone... > > > > > >>> > > > > > >>> I just updated the wiki for KIP-32. > > > > > >>> > > > > > >>> Gwen, > > > > > >>> > > > > > >>> Yes, the migration plan is what you described. > > > > > >>> > > > > > >>> I agree with your comments on the version. > > > > > >>> I changed message.format.version to use the release version. > > > > > >>> I did not change the internal version, we can discuss this in= a > > > > > separate > > > > > >>> thread. > > > > > >>> > > > > > >>> Thanks, > > > > > >>> > > > > > >>> Jiangjie (Becket) Qin > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>>> On Dec 24, 2015, at 5:38 AM, Guozhang Wang < > wangguoz@gmail.com> > > > > > wrote: > > > > > >>>> > > > > > >>>> Also I agree with Gwen that such changes may worth a 0.10 > > release > > > or > > > > > >> even > > > > > >>>> 1.0, having it in 0.9.1 would be quite confusing to users. > > > > > >>>> > > > > > >>>> Guozhang > > > > > >>>> > > > > > >>>>> On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang < > > > wangguoz@gmail.com > > > > > > > > > > >>> wrote: > > > > > >>>>> > > > > > >>>>> Becket, > > > > > >>>>> > > > > > >>>>> Please let us know once you have updated the wiki page > > regarding > > > > the > > > > > >>>>> migration plan. Thanks! > > > > > >>>>> > > > > > >>>>> Guozhang > > > > > >>>>> > > > > > >>>>>> On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira < > > > gwen@confluent.io > > > > > > > > > > >>> wrote: > > > > > >>>>>> > > > > > >>>>>> Thanks Becket, Anne and Neha for responding to my concern. > > > > > >>>>>> > > > > > >>>>>> I had an offline discussion with Anne where she helped me > > > > understand > > > > > >>> the > > > > > >>>>>> migration process. It isn't as bad as it looks in the KIP = :) > > > > > >>>>>> > > > > > >>>>>> If I understand it correctly, the process (for users) will > be: > > > > > >>>>>> > > > > > >>>>>> 1. Prepare for upgrade (set format.version =3D 0, ApiVersi= on =3D > > > > 0.9.0) > > > > > >>>>>> 2. Rolling upgrade of brokers > > > > > >>>>>> 3. Bump ApiVersion to 0.9.0-1, so fetch requests between > > brokers > > > > > will > > > > > >>> use > > > > > >>>>>> the new protocol > > > > > >>>>>> 4. Start upgrading clients > > > > > >>>>>> 5. When "enough" clients are upgraded, bump format.version > to > > 1 > > > > > >>> (rolling). > > > > > >>>>>> > > > > > >>>>>> Becket, can you confirm? > > > > > >>>>>> > > > > > >>>>>> Assuming this is the process, I'm +1 on the change. > > > > > >>>>>> > > > > > >>>>>> Reminder to coders and reviewers that pull-requests with > > > > user-facing > > > > > >>>>>> changes should include documentation changes as well as co= de > > > > > changes. > > > > > >>>>>> And a polite request to try to be helpful to users on when > to > > > use > > > > > >>>>>> create-time and when to use log-append-time as > configuration - > > > > this > > > > > >> is > > > > > >>> not > > > > > >>>>>> a trivial decision. > > > > > >>>>>> > > > > > >>>>>> A separate point I'm going to raise in a different thread = is > > > that > > > > we > > > > > >>> need > > > > > >>>>>> to streamline our versions a bit: > > > > > >>>>>> 1. I'm afraid that 0.9.0-1 will be confusing to users who > care > > > > about > > > > > >>>>>> released versions (what if we forget to change it before t= he > > > > > release? > > > > > >>> Is > > > > > >>>>>> it > > > > > >>>>>> meaningful enough to someone running off trunk?), we need = to > > > come > > > > up > > > > > >>> with > > > > > >>>>>> something that will work for both LinkedIn and everyone > else. > > > > > >>>>>> 2. ApiVersion has real version numbers. > message.format.version > > > has > > > > > >>>>>> sequence > > > > > >>>>>> numbers. This makes us look pretty silly :) > > > > > >>>>>> > > > > > >>>>>> My version concerns can be addressed separately and should > not > > > > hold > > > > > >>> back > > > > > >>>>>> this KIP. > > > > > >>>>>> > > > > > >>>>>> Gwen > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> On Tue, Dec 22, 2015 at 11:01 PM, Becket Qin < > > > > becket.qin@gmail.com> > > > > > >>>>>> wrote: > > > > > >>>>>> > > > > > >>>>>>> Hi Anna, > > > > > >>>>>>> > > > > > >>>>>>> Thanks for initiating the voting process. I did not start > the > > > > > voting > > > > > >>>>>>> process because there were still some ongoing discussion > with > > > Jun > > > > > >>> about > > > > > >>>>>> the > > > > > >>>>>>> timestamp regarding compressed messages. That is why the > wiki > > > > page > > > > > >>>>>> hasn't > > > > > >>>>>>> reflected the latest conversation as Guozhang pointed out= . > > > > > >>>>>>> > > > > > >>>>>>> Like Neha said I think we have reached general agreement = on > > > this > > > > > >> KIP. > > > > > >>> So > > > > > >>>>>>> it is probably fine to start the KIP voting. At least we > draw > > > > more > > > > > >>>>>>> attention to the KIP even if there are some new discussio= n > to > > > > bring > > > > > >>> up. > > > > > >>>>>>> > > > > > >>>>>>> Regarding the upgrade plan, given we decided to implement > > > KIP-31 > > > > > and > > > > > >>>>>>> KIP-32 in the same patch to avoid change binary protocol > > twice, > > > > the > > > > > >>>>>> upgrade > > > > > >>>>>>> plan was mostly discussed on the discussion thread of > KIP-31. > > > > > >>>>>>> > > > > > >>>>>>> Since the voting has been initiated, I will update the wi= ki > > > with > > > > > >>> latest > > > > > >>>>>>> conversation to avoid further confusion. > > > > > >>>>>>> > > > > > >>>>>>> BTW, I actually have started coding work on KIP-31 and > KIP-32 > > > and > > > > > >> will > > > > > >>>>>>> focus on the patch before I return from vacation in mid J= an > > > > because > > > > > >> I > > > > > >>>>>> have > > > > > >>>>>>> no LInkedIn VPN access in China anyway... > > > > > >>>>>>> > > > > > >>>>>>> Thanks, > > > > > >>>>>>> > > > > > >>>>>>> Jiangjie > > > > > >>>>>>> > > > > > >>>>>>>> On Dec 23, 2015, at 12:31 PM, Anna Povzner < > > anna@confluent.io > > > > > > > > > >>> wrote: > > > > > >>>>>>>> > > > > > >>>>>>>> Hi Gwen, > > > > > >>>>>>>> > > > > > >>>>>>>> I just wanted to point out that I just started the vote. > > > Becket > > > > > >> wrote > > > > > >>>>>> the > > > > > >>>>>>>> proposal and led the discussions. > > > > > >>>>>>>> > > > > > >>>>>>>> What I understood from reading the discussion thread, th= e > > > > > migration > > > > > >>>>>> plan > > > > > >>>>>>>> was discussed at the KIP meeting, and not much on the > > mailing > > > > list > > > > > >>>>>>> itself. > > > > > >>>>>>>> > > > > > >>>>>>>> My question about the migration plan was same as Guozhan= g > > > wrote: > > > > > >> The > > > > > >>>>>> case > > > > > >>>>>>>> when an upgraded broker receives an old producer request= . > > The > > > > > >>>>>> proposal is > > > > > >>>>>>>> for the broker to fill in the timestamp field with the > > current > > > > > time > > > > > >>> at > > > > > >>>>>>> the > > > > > >>>>>>>> broker. Cons: it goes against the definition of CreateTi= me > > > type > > > > of > > > > > >>> the > > > > > >>>>>>>> timestamp (we are "over-writing" it at the broker). Pros= : > It > > > > looks > > > > > >>>>>> like > > > > > >>>>>>>> most of the use-cases would actually want that behavior, > > > because > > > > > >>>>>>> otherwise > > > > > >>>>>>>> timestamp is useless and they will need to support an ol= d, > > > > > >>>>>> pre-timestamp, > > > > > >>>>>>>> behavior. E.g., if we modify log retention policy to use > the > > > > > >>>>>> timestamp, > > > > > >>>>>>> we > > > > > >>>>>>>> would need to support an old implementation (the one tha= t > > does > > > > not > > > > > >>> use > > > > > >>>>>>>> timestamps in the message). So I actually have a > preference > > > for > > > > > the > > > > > >>>>>>>> proposed approach. > > > > > >>>>>>>> > > > > > >>>>>>>> Thanks, > > > > > >>>>>>>> Anna > > > > > >>>>>>>> > > > > > >>>>>>>>> On Tue, Dec 22, 2015 at 8:02 PM, Neha Narkhede < > > > > > neha@confluent.io > > > > > >>> > > > > > >>>>>>> wrote: > > > > > >>>>>>>>> > > > > > >>>>>>>>> Hey Gwen, > > > > > >>>>>>>>> > > > > > >>>>>>>>> Migration plan wasn't really discussed a ton in the > > previous > > > > > >>> threads. > > > > > >>>>>>> So it > > > > > >>>>>>>>> will be great to dive deep and see if there are gaps > > there. I > > > > had > > > > > >>>>>> some > > > > > >>>>>>>>> questions, but the details listed on the KIP are great. > > > > > >>>>>>>>> > > > > > >>>>>>>>> It is complex, though the plan outlined in the wiki > > assumes a > > > > > zero > > > > > >>>>>>> downtime > > > > > >>>>>>>>> upgrade assuming that producers and consumers can't be > > > upgraded > > > > > in > > > > > >>>>>>> tandem. > > > > > >>>>>>>>> This is typical for companies that have a significant > Kafka > > > > > >>>>>> footprint, > > > > > >>>>>>> like > > > > > >>>>>>>>> LinkedIn. > > > > > >>>>>>>>> > > > > > >>>>>>>>> Thanks, > > > > > >>>>>>>>> Neha > > > > > >>>>>>>>> > > > > > >>>>>>>>>> On Tue, Dec 22, 2015 at 7:48 PM, Gwen Shapira < > > > > > gwen@confluent.io > > > > > >>> > > > > > >>>>>>> wrote: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Hi Anna, > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Thanks for the KIP, especially for the details on all > the > > > > > >>>>>> alternatives > > > > > >>>>>>>>> and > > > > > >>>>>>>>>> how we arrived at the proposal. Its really great! > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Can you point me at where the migration plan was > > discussed? > > > It > > > > > >>> looks > > > > > >>>>>>>>> overly > > > > > >>>>>>>>>> complex and I have a bunch of questions, but if there > was > > a > > > > > >>>>>> discussion, > > > > > >>>>>>>>> I'd > > > > > >>>>>>>>>> like to read up rather than repeating it :) > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Gwen > > > > > >>>>>>>>>> > > > > > >>>>>>>>>>> On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner < > > > > > >> anna@confluent.io > > > > > >>>> > > > > > >>>>>>>>>> wrote: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>>> Hi, > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> I am opening the voting thread for KIP-32: Add > CreateTime > > > and > > > > > >>>>>>>>>>> LogAppendTime to Kafka message. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> For reference, here's the KIP wiki: > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime= +and+LogAppendTime+to+Kafka+message > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> And the mailing list threads: > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> September: > > > > > >> > > > > > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm= 6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> October: > > > > > >> > > > > > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm= 7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> December: > > > > > >> > > > > > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm= 4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Thanks > > > > > >>>>>>>>>>> Anna > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> > > > > > >>>>>>>>> -- > > > > > >>>>>>>>> Thanks, > > > > > >>>>>>>>> Neha > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> -- > > > > > >>>>> -- Guozhang > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> -- > > > > > >>>> -- Guozhang > > > > > >> > > > > > > > > > > > > > > > --001a113d216eb20cde0528d9dd06--