Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 91F49200BD4 for ; Fri, 16 Dec 2016 20:09:53 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 906D7160B24; Fri, 16 Dec 2016 19:09:53 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3F044160B10 for ; Fri, 16 Dec 2016 20:09:51 +0100 (CET) Received: (qmail 65810 invoked by uid 500); 16 Dec 2016 19:09:50 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 65794 invoked by uid 99); 16 Dec 2016 19:09:49 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Dec 2016 19:09:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 1546B181260 for ; Fri, 16 Dec 2016 19:09:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.479 X-Spam-Level: ** X-Spam-Status: No, score=2.479 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=confluent-io.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id FkYdc97v3jfv for ; Fri, 16 Dec 2016 19:09:28 +0000 (UTC) Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com [74.125.82.45]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 1EC295FAF3 for ; Fri, 16 Dec 2016 19:09:28 +0000 (UTC) Received: by mail-wm0-f45.google.com with SMTP id f82so46325910wmf.1 for ; Fri, 16 Dec 2016 11:09:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=confluent-io.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=Fk1ku4yKTIOgnIJ76H0FvgquYaQTEjkoIs/RLXH7o6A=; b=bmUsbZ5XDRTzykbCtQXOWJPOiupTtIQeuafe7lh0XsR80lZPFWXeFgCbUlMYILOhNb iX9cigekOPeY5qmqh9HNTv3YQrdfBELGq1JhQn+DFxXIVVS519XHoxUnEl6fNkly6yEy hRZJIRCIE9J47pHVswmCqcafS6FXHeeNsTH1Q5mS0R4jRYcd19n7m4x3ujaWkQCO+b8V kqi7azft0dJr5ajWEefbv1rLaynNWkJs2HtusYAtl6SQKeja7ttQyuwW4V0/X6AT0qf6 iUlh6BFSg5PPzZds/FUNlhZ56oFlkIBf/lmlY3ZYppsYNr1pC54rkfOidcjlLMDisCQJ kHvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=Fk1ku4yKTIOgnIJ76H0FvgquYaQTEjkoIs/RLXH7o6A=; b=F3mV4iD/fz9cA4tUjpXSGIbTmY3y29UKdbvaIpCHqvtYGowLmmr/hFbPM+YxPqZ5Xj rzgmXnz1WdlisJP9LtdVEBJURQk42PmamJ8W11Dl2TT7E5Bm7+CPungIxRZlekseMEHF G+FPZ0FbT+4fzQ0koTGxZUyCNJHXz+l4zCRaVa1v+IAeYgoJPhTcCYHIIwoLTfR6A2i2 o8+UdqsftEKV9Qhy5ce8Yt0BS4qRSFAAU9nJdRF8tEbqeFl8UXzAfOKo2XMdMwiCCy5u ekfp9RUJosNiapzJWNmuPu/HiGRr8CsThcr//xQCmP2DcYcX/vXqXP6yIoT6853+wWG2 REhw== X-Gm-Message-State: AIkVDXLyGtGBjyUfYa+JHJFiEFo0S2Z5txTdkiSyPOs8YpkpdQRjN1YCizBCr6mYA9mktkKt X-Received: by 10.28.189.69 with SMTP id n66mr4296926wmf.35.1481915367065; Fri, 16 Dec 2016 11:09:27 -0800 (PST) Received: from mail-wj0-f180.google.com (mail-wj0-f180.google.com. [209.85.210.180]) by smtp.gmail.com with ESMTPSA id 63sm4655855wmg.2.2016.12.16.11.09.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 16 Dec 2016 11:09:26 -0800 (PST) Received: by mail-wj0-f180.google.com with SMTP id tg4so100224118wjb.1 for ; Fri, 16 Dec 2016 11:09:25 -0800 (PST) X-Received: by 10.194.65.74 with SMTP id v10mr4557727wjs.168.1481915365130; Fri, 16 Dec 2016 11:09:25 -0800 (PST) MIME-Version: 1.0 Received: by 10.28.14.195 with HTTP; Fri, 16 Dec 2016 11:09:23 -0800 (PST) In-Reply-To: <076409F4-D89B-4D43-B75D-8C43A43FCB85@ig.com> References: <7AD186ED-7982-4A5E-920B-5C8AAE5084BF@ig.com> <2ACDC47F-DB41-45A1-B62A-97B805A1D8C4@ig.com> <59bba6729002497a9dbae45d1da59241@BMPRDEXC142.IGI.IG.LOCAL> <46264a410f8c4aa48ff7caef0bb1ecd3@BMPRDEXC142.IGI.IG.LOCAL> <1481433852678.26033@ig.com> <1481455584685.73053@ig.com> <42c21cfc4cb24b7a908092b734f5ca52@BMPRDEXC142.IGI.IG.LOCAL> <8a332f08017f4bbf8aba4bcd34fd44a8@BMPRDEXC142.IGI.IG.LOCAL> <076409F4-D89B-4D43-B75D-8C43A43FCB85@ig.com> From: Jay Kreps Date: Fri, 16 Dec 2016 11:09:23 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag To: "dev@kafka.apache.org" Content-Type: multipart/alternative; boundary=047d7bd91e9eed11bf0543cb4e6f archived-at: Fri, 16 Dec 2016 19:09:53 -0000 --047d7bd91e9eed11bf0543cb4e6f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hey Michael, I do think it might have been better had we started with a separate concept of null vs delete. Given that we are where we are, I'm not sure that the possible cures we explored so far are better than the disease. I apologize for coming to this late, but I didn't really understand the implications of the KIP and that we'd be breaking compatibility with existing apps until the vote had begun, and in my defense I'm not sure the other folks voting did either. I think we all agree there are many existing apps that are built with the assumption of "null value non-tombstone" and it isn't possible to disambiguate these from tombstones on the producer. It isn't that anyone is saying we have to support all four possibilities at once, it's that we simply can't orphan one of the existing combinations or our users will eat us! If I've understood your alternate solution of adding another setting for compaction, I think this does fix the compatibility problem, but it adds an odd mode the user has to add on all their topics. While the current state is easily explainable, the resulting state where the meaning of tombstone and null are overlapping and ambiguous and dependent on a compaction setting that could change out of band or not be in sync with your code in some environment seems worse to me then where we currently are. I think the question is how would this combination be explained to users and does it make sense? -Jay On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce wrote: > Hi Chaps, > > Can we either get one more +1 binding (we have 2 already) on the existing= ? > > Or have some response on the below possible alternative. We are keen to > get working on this, so we make next feature release. > > Cheers > Mike > > > On 13/12/2016, 16:32, "Michael Pearce" wrote: > > Hi Ismael > > Did you see our email this morning, what's your thoughts on this > approach to instead we simply have a brand new policy? > > Cheers > Mike > > > Sent using OWA for iPhone > ________________________________________ > From: ismaelj@gmail.com on behalf of Ismael Juma = < > ismael@juma.me.uk> > Sent: Tuesday, December 13, 2016 11:30:05 AM > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag > > Yes, this is actually tricky to do in a way where we both have the > desired > semantics and maintain compatibility. When someone creates a > `ProducerRecord` with a `null` value today, the producer doesn't know > if > it's meant to be a tombstone or not. For V3 messages, it's easy when > the > constructor that takes a tombstone is used. However, if any other > constructor is used, it's not clear. A couple of options I can think > of, > none of them particularly nice: > > 1. Have a third state where tombstone =3D unknown and the broker woul= d > set > the tombstone bit if the value was null and the topic was compacted. > People > that wanted to pass a non-null value for the tombstone would have to > use > the constructor that takes a tombstone. The drawbacks: third state fo= r > tombstone in message format, message conversion at the broker for a > common > case. > > 2. Extend MetadataResponse to optionally include topic configs, which > would > make it possible for the producer to be smarter about setting the > tombstone. It would only do it if a tombstone was not passed > explicitly, > the value was null and the topic was compacted. The main drawback is > that > the producer would be getting a bit more data for each topic even > though it > probably won't use it most of the time. Extending MetadataResponse to > return topic configs would be useful for other reasons as well, so th= at > part seems OK. > > In addition, for both proposals, we could consider adding warnings to > the > documentation that the behaviour of the constructors that don't take = a > tombstone would change in the next major release so that tombstone = =3D > false. > Not sure if this would be worth it though. > > Ismael > > On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava < > ewen@confluent.io> > wrote: > > > Michael, > > > > It kind of depends on how you want to interpret the tombstone flag. > If it's > > purely a producer-facing Kafka-level thing that we treat as interna= l > to the > > broker and log cleaner once the record is sent, then your approach > makes > > sense. You're just moving copying the null-indicates-delete behavio= r > of the > > old constructor into the tombstone flag. > > > > However, if you want this change to more generally decouple the ide= a > of > > deletion and null values, then you are sometimes converting what > might be a > > completely valid null value that doesn't indicate deletion into a > > tombstone. Downstream applications could potentially handle these > cases > > differently given the separation of deletion from value. > > > > I guess the question is if we want to try to support the latter eve= n > for > > topics where we have older produce requests. An example where this > could > > come up is in something like a CDC Connector. If we try to support > the > > semantic difference, a connector might write changes to Kafka using > the > > tombstone flag to indicate when a row was truly deleted (vs an > update that > > sets it to null but still present; this probably makes more sense > for CDC > > from document stores or extracting single columns). There are vario= us > > reasons we might want to maintain the full log and not turn > compaction on > > (or just use a time-based retention policy), but downstream > applications > > might care to know the difference between a delete and a null value= . > In > > fact both versions of the same log (compacted and time-retention) > could be > > useful and I don't think it'll be uncommon to maintain both or use > KIP-71 > > to maintain a hybrid compacted/retention topic. > > > > -Ewen > > > > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce < > Michael.Pearce@ig.com> > > wrote: > > > > > Hi Jay, > > > > > > Why wouldn't that work, the tombstone value is only looked at by > the > > > broker, on a topic configured for compaction as such is benign on > non > > > compacted topics. This is as much as sending a null value current= ly > > > > > > > > > Regards > > > Mike > > > > > > > > > > > > Sent using OWA for iPhone > > > ________________________________________ > > > From: Jay Kreps > > > Sent: Sunday, December 11, 2016 8:58:53 PM > > > To: dev@kafka.apache.org > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag > > > > > > Hey Michael, > > > > > > I'm not quite sure that works as that would translate ALL null > values to > > > tombstones, even for non-compacted topics that use null as an > acceptable > > > value sent by the producer and expected by the consumer. > > > > > > -Jay > > > > > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce < > Michael.Pearce@ig.com> > > > wrote: > > > > > > > Hi Ewen, > > > > > > > > I think the easiest way to show this is with code. > > > > > > > > As you can see we keep the existing behaviour for code/binaries > calling > > > > the pre-existing constructors, whereby if the value is null the > > tombstone > > > > is set to true. > > > > > > > > Regards > > > > Mike > > > > > > > > > > > > > > > > /** > > > > * Creates a record with a specified timestamp to be sent t= o > a > > > > specified topic and partition > > > > * > > > > * @param topic The topic the record will be appended to > > > > * @param partition The partition to which the record shoul= d > be > > sent > > > > * @param timestamp The timestamp of the record > > > > * @param tombstone if the record should be treated as a > tombstone > > if > > > > the topic is compacted > > > > * @param key The key that will be included in the record > > > > * @param value The record contents > > > > */ > > > > public ProducerRecord(String topic, Integer partition, > Boolean > > > > tombstone, Long timestamp, K key, V value) { > > > > if (topic =3D=3D null) > > > > throw new IllegalArgumentException("Topic cannot be > > null."); > > > > if (timestamp !=3D null && timestamp < 0) > > > > throw new IllegalArgumentException( > > > > String.format("Invalid timestamp: %d. > Timestamp > > > should > > > > always be non-negative or null.", timestamp)); > > > > if (partition !=3D null && partition < 0) > > > > throw new IllegalArgumentException( > > > > String.format("Invalid partition: %d. > Partition > > > number > > > > should always be non-negative or null.", partition)); > > > > if (!tombstone && value =3D=3D null){ > > > > throw new IllegalArgumentException( > > > > String.format("Tombstone must be true if nu= ll > > > value")); > > > > } > > > > this.topic =3D topic; > > > > this.partition =3D partition; > > > > this.tombstone =3D tombstone; > > > > this.key =3D key; > > > > this.value =3D value; > > > > this.timestamp =3D timestamp; > > > > } > > > > > > > > public ProducerRecord(String topic, Integer partition, Long > > > timestamp, > > > > K key, V value) { > > > > this(topic, partition, value =3D=3D null, timestamp, ke= y, > value); > > > > } > > > > ________________________________________ > > > > From: Ewen Cheslack-Postava > > > > Sent: Sunday, December 11, 2016 5:45 AM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag > > > > > > > > It seemed like this was addressed in the migration section, > wasn't it? > > > The > > > > V2 vs V3 requests and the fact that the broker will downgrade t= he > > message > > > > format (losing zero copy) if you issues a V2 request to a broke= r > using > > V3 > > > > format handles compatibility, does it not? The existing > consumers won't > > > see > > > > the extra metadata in the value, but they will get a null > instead and > > > treat > > > > it as a tombstone. Certainly there is a performance impact, but > it > > seemed > > > > compatible. > > > > > > > > I'm worried about this though: > > > > > > > > > > > > - *NOTE *: With the new version of producer client using > > > ProduceRequest > > > > V3 (magic byte =3D 2), a non tombstone (tombstone bit not se= t) > and > > null > > > > value > > > > should not be allowed as the older version of consumer using > > > > FetchRequest > > > > V2 will think of this as a tombstone when its actually not. > > > > > > > > > > > > Unless I'm misunderstanding, this ends up breaking binary > compatibility > > > for > > > > the Java API. It sounds like this suggests that passing a null > value to > > > the > > > > existing ProducerRecord constructors would generate an exceptio= n > since > > > you > > > > didn't explicitly enable the tombstone (via whatever new > constructor is > > > > provided). But that means you can't swap in a newer client jar > without > > > > recompiling and get the same behavior if your app deletes keys > using > > the > > > > current approach because your app will start throwing > exceptions. Maybe > > > > this is a tradeoff we're ok with, but we've tried pretty hard t= o > avoid > > > > breaking compatibility like this so far -- it makes picking up > bug > > fixes > > > in > > > > the clients harder for users of frameworks that have to pin to > earlier > > > > default versions for compatibility. > > > > > > > > But then later the KIP says: > > > > > > > > > > > > - The old constructors for ProducerRecord without this > parameter > > will > > > be > > > > kept but updated so that their default behaviour if setting > null > > value > > > > will > > > > be the tombstone will be set to true to keep existing > behaviour. > > > > > > > > > > > > So maybe I am misinterpreting something. > > > > > > > > And just a nit re: motivation section, item 6 would be clearer > for a > > > union > > > > schema with null (or optional schemas in other formats), e.g. > [null, > > > > string], in which case losing the schema truly is losing > information > > > > (whereas null is already the only valid value for a pure null > schema). > > > > > > > > -Ewen > > > > > > > > > > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce < > Michael.Pearce@ig.com > > > > > > > wrote: > > > > > > > > > Hi Jay, > > > > > > > > > > Good point this detail is missing in the KIP write up. Ive > added this > > > > now. > > > > > > > > > > Essentially simply just upgrading the clients we do not expec= t > any > > > client > > > > > app code change needed. > > > > > > > > > > Cheers > > > > > Mike > > > > > ________________________________________ > > > > > From: Jay Kreps > > > > > Sent: Saturday, December 10, 2016 10:51 PM > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag > > > > > > > > > > Michael, > > > > > > > > > > The compatibility section goes through the migration path, bu= t > isn't > > > the > > > > > bigger compatibility issue with existing apps? There are many > > (probably > > > > > thousands) of apps in production that use this feature and > send null > > to > > > > > mean delete. It seems like this would break compatibility wit= h > them, > > > and > > > > > they would have to be rewritten to right? > > > > > > > > > > -Jay > > > > > > > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce < > > Michael.Pearce@ig.com > > > > > > > > > wrote: > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > 4) On v3 we honour the tombstone. As such we expect it to b= e > set > > > > > correctly > > > > > > as per the KIP. > > > > > > > > > > > > 4.1) why would we want to produce an error when its v3? Thi= s > is the > > > > exact > > > > > > purpose to support non-null tombstone=E2=80=99s > > > > > > 4.2) again here im unclear on the question on the v3, produ= ce > > request > > > > we > > > > > > expect the tombstone flag to be set correctly. > > > > > > > > > > > > 4.4) compaction only occurs on compacted topics, the bit > makes no > > > > > > difference and not looked at on un-compacted (time/size bas= ed > > > > eviction). > > > > > > > > > > > > > > > > > > On 06/12/2016, 20:08, "Jun Rao" wrote: > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > 4. Then, I think I misunderstood this point. Could you > document > > > the > > > > > > following points in the wiki? > > > > > > 4.1 If producer V3 sets tombstone, but provides a > non-null > > value, > > > > > does > > > > > > the > > > > > > send() get an error or does the producer automatically > set the > > > > value > > > > > to > > > > > > null? > > > > > > 4.2 If producer V3 doesn't set tombstone, but provides = a > null > > > > value, > > > > > > does > > > > > > the send() get an error or does the producer > automatically sets > > > the > > > > > > tombstone? > > > > > > 4.3 Does the broker only expect messages that (a) have = no > > > tombstone > > > > > and > > > > > > non-null value; (b) have tombstone and null value and > reject > > the > > > > > > messages > > > > > > with an error code otherwise? > > > > > > 4.4 Do 4.1, 4.2, 4.3 depend on whether the topic is > compacted > > on > > > > > not? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce < > > > > > Michael.Pearce@ig.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Not at all. This only acts on compacted topics just > as what > > > > occurs > > > > > > today > > > > > > > > > > > > > > Sent using OWA for iPhone > > > > > > > ________________________________________ > > > > > > > From: Jun Rao > > > > > > > Sent: Tuesday, December 6, 2016 6:25:28 PM > > > > > > > To: dev@kafka.apache.org > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone > Flag > > > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > > > 4. Hmm, does that mean the new client library can > never send > > a > > > > null > > > > > > message > > > > > > > even to a regular topic? This seems like a change of > the > > > existing > > > > > > behavior. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce < > > > > > > Michael.Pearce@ig.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > > > Re 4) That's because we expect the tombstone value > to be > > set > > > > > > correctly if > > > > > > > > message bit is 2, as such if an older client sends > in on > > old > > > > > > message the > > > > > > > > message is upcast and the bit is set correctly. And > such no > > > > > longer > > > > > > need > > > > > > > to > > > > > > > > check the value. Mayuresh can you confirm my > thinking and > > > > > > understanding > > > > > > > of > > > > > > > > what we've discussed? > > > > > > > > > > > > > > > > The second point I understand what you're getting a= t > now my > > > > > > apologies. > > > > > > > Yes > > > > > > > > this makes sense to save on touching the message, i= f > we're > > > the > > > > > > only kip > > > > > > > > going in, in this release. > > > > > > > > > > > > > > > > Cheers > > > > > > > > Mike > > > > > > > > > > > > > > > > Sent using OWA for iPhone > > > > > > > > ________________________________________ > > > > > > > > From: Jun Rao > > > > > > > > Sent: Tuesday, December 6, 2016 5:22:13 PM > > > > > > > > To: dev@kafka.apache.org > > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction > Tombstone Flag > > > > > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > > > > > 4. Is this updated in the wiki? The text "If the > magic byte > > > on > > > > > > message is > > > > > > > > 2, the broker should use the tombstone bit for log > > > compaction." > > > > > > doesn't > > > > > > > > seem to have changed. > > > > > > > > > > > > > > > > 2. My point is that if we change the message format > just > > for > > > > this > > > > > > KIP, we > > > > > > > > should consider whether it's worth optimizing the > down > > > > conversion > > > > > > path > > > > > > > > (i.e., decide whether a conversion is needed by jus= t > > looking > > > at > > > > > the > > > > > > > > tombstone bit in the wrapper message) since > tombstone will > > be > > > > > used > > > > > > > rarely. > > > > > > > > However, if the message format change here is > combined with > > > > other > > > > > > KIPs, > > > > > > > > then this optimization likely won't be needed. The > latter > > > > > probably > > > > > > makes > > > > > > > > the code simpler. Jiangjie, Mayuresh, what do you > think? > > > > > > > > > > > > > > > > Other than those, +1 from me, > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce < > > > > > > Michael.Pearce@ig.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Jun > > > > > > > > > > > > > > > > > > do we have your vote on this now? > > > > > > > > > > > > > > > > > > Any other concerns? > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > Sent using OWA for iPhone > > > > > > > > > ________________________________________ > > > > > > > > > From: Michael Pearce > > > > > > > > > Sent: Saturday, December 3, 2016 1:37:45 AM > > > > > > > > > To: dev@kafka.apache.org > > > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction > Tombstone > > Flag > > > > > > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > > > > > Have updated. Thanks again for the feedback. > > > > > > > > > > > > > > > > > > Agree yes we should align up when it gets to that= , > I > > assume > > > > > > you=E2=80=99ve > > > > > > > > flagged > > > > > > > > > the same in the other KIP? > > > > > > > > > > > > > > > > > > I think for now let=E2=80=99s get this KIP approv= ed, then > we can > > > > start > > > > > > the work > > > > > > > > to > > > > > > > > > get an initial PR, then we can discuss how to > align the > > two > > > > if > > > > > > needed. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > On 02/12/2016, 18:26, "Jun Rao" > > wrote: > > > > > > > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > > > > > > > For 2), this is fine. Could you update the KI= P > wiki > > to > > > > make > > > > > > this > > > > > > > and > > > > > > > > > other > > > > > > > > > points clearer? Other than that, the KIP look= s > good > > to > > > > me. > > > > > > > > > > > > > > > > > > An orthogonal thing is that there are other > KIPs such > > > as > > > > > > KIP-98 > > > > > > > that > > > > > > > > > also > > > > > > > > > intend to change the message format. If they > all get > > > > > > approved, we > > > > > > > > > should > > > > > > > > > think about whether it's better to just bump > up the > > > magic > > > > > > byte once > > > > > > > > to > > > > > > > > > incorporate multiple format changes like we > did in > > > > > > KIP-31/KIP-32. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearc= e > < > > > > > > > > Michael.Pearce@ig.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Jao > > > > > > > > > > > > > > > > > > > > Thanks for the response. Sorry for slow > reply, both > > > > with > > > > > > personal > > > > > > > > > sickness > > > > > > > > > > and also battling some critical issues > encountered > > > > since > > > > > > > upgrading > > > > > > > > to > > > > > > > > > > 0.10.1.0 > > > > > > > > > > > > > > > > > > > > 1) Thans for spotting, Document error where > we > > > branched > > > > > > this KIP > > > > > > > > from > > > > > > > > > > KIP-82, will get that removed. > > > > > > > > > > 2) Intent is to do this just at the record > message > > > > level. > > > > > > > > > > 3) Thanks for spotting, Will ensure this is > > > corrected. > > > > > > > > > > 4) As per discussion thread we will support > > > tombstone + > > > > > > null > > > > > > > value, > > > > > > > > > > tombstone + non null value, no tombstone + > null > > > value. > > > > > > > > > > 5) I believe this was in the discussion > thread, > > > > @Mayuresh > > > > > > is this > > > > > > > > > > something we=E2=80=99ve overlooked? I thoug= ht we > would down > > > > > > convert and > > > > > > > > > remove the > > > > > > > > > > value so the old consumer had existing > behavior, or > > > is > > > > > > there > > > > > > > > > something we > > > > > > > > > > haven=E2=80=99t thought about? > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > On 30/11/2016, 18:12, "Jun Rao" < > jun@confluent.io> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi, Michael, > > > > > > > > > > > > > > > > > > > > Thanks for the KIP. A few comments belo= w. > > > > > > > > > > > > > > > > > > > > 1. The message format change contains > > > > "HeadersLength > > > > > > > Headers". > > > > > > > > > Is that > > > > > > > > > > intended? > > > > > > > > > > > > > > > > > > > > 2. For compressed messageset, is the > tombstone > > > bit > > > > > > only set > > > > > > > at > > > > > > > > > the > > > > > > > > > > shallow > > > > > > > > > > level? Do we always leave that bit in t= he > > wrapper > > > > > > message > > > > > > > > unset? > > > > > > > > > An > > > > > > > > > > alternative is to set the tombstone bit > in the > > > > > wrapper > > > > > > if at > > > > > > > > > least one > > > > > > > > > > inner message has the tombstone bit set= . > This > > > makes > > > > > > things a > > > > > > > > bit > > > > > > > > > more > > > > > > > > > > complicated, but we could potentially > exploit > > > that > > > > > for > > > > > > > > > optimizing down > > > > > > > > > > conversion. For example, we only need t= o > > convert > > > > > > messages > > > > > > > with > > > > > > > > > magic 2 > > > > > > > > > > to > > > > > > > > > > magic 1 if the wrapper's tombstone bit > is set > > > > > > (conversion is > > > > > > > > > always > > > > > > > > > > needed > > > > > > > > > > from magic 2 to magic 0). Not sure if t= he > > > > > optimization > > > > > > is > > > > > > > worth > > > > > > > > > the > > > > > > > > > > complexity though. > > > > > > > > > > > > > > > > > > > > 3. The referencing of the new version o= f > > > > > > > > > ProducerRequest/FetchRequest > > > > > > > > > > is > > > > > > > > > > inconsistent (v4 vs v3). Since our > convention > > > > starts > > > > > at > > > > > > > version > > > > > > > > > at 0, I > > > > > > > > > > think the new version would be 3. > > > > > > > > > > > > > > > > > > > > 4. "If the magic byte on message is 2, > the > > broker > > > > > > should use > > > > > > > > the > > > > > > > > > > tombstone > > > > > > > > > > bit for log compaction." What about nul= l > value? > > > My > > > > > > > > understanding > > > > > > > > > is > > > > > > > > > > that > > > > > > > > > > null value will be treated the same as > setting > > > the > > > > > > tombstone > > > > > > > > bit. > > > > > > > > > > > > > > > > > > > > 5. For the migration path, it would be > useful > > to > > > > > > describe the > > > > > > > > > down > > > > > > > > > > conversion path to consumers (i.e., > brokers on > > > > > message > > > > > > format > > > > > > > > > 0.10.2 > > > > > > > > > > and > > > > > > > > > > consumers on older version). > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 29, 2016 at 3:18 AM, Michae= l > > Pearce < > > > > > > > > > Michael.Pearce@ig.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > We have been discussing in the below > thread > > and > > > > > final > > > > > > > changes > > > > > > > > > have > > > > > > > > > > been > > > > > > > > > > > made to the KIP wiki based on these > > > discussions. > > > > > > > > > > > > > > > > > > > > > > We would now like to put to the vote > the > > > > following > > > > > > KIP: > > > > > > > > > > > https://cwiki.apache.org/ > > > > > > confluence/display/KAFKA/KIP- > > > > > > > 87+-+ > > > > > > > > > > > Add+Compaction+Tombstone+Flag > > > > > > > > > > > > > > > > > > > > > > This kip is for having a distinct > compaction > > > > > > attribute > > > > > > > > > =E2=80=9Ctombstone=E2=80=9D > > > > > > > > > > flag > > > > > > > > > > > instead of relying on null value, > allowing > > > > non-null > > > > > > value > > > > > > > > > delete > > > > > > > > > > messages. > > > > > > > > > > > > > > > > > > > > > > Many thanks, > > > > > > > > > > > Michael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 22/11/2016, 15:52, "Michael Pearce= " > < > > > > > > > > Michael.Pearce@ig.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi Mayuresh, > > > > > > > > > > > > > > > > > > > > > > LGTM. Ive just made one small > adjustment > > > > > > updating the > > > > > > > > wire > > > > > > > > > > protocol to > > > > > > > > > > > show the magic byte bump. > > > > > > > > > > > > > > > > > > > > > > Do we think we=E2=80=99re good to= put to a > vote? > > Is > > > > > > there any > > > > > > > > > other bits > > > > > > > > > > > needing discussion? > > > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > On 21/11/2016, 18:26, "Mayuresh > Gharat" < > > > > > > > > > > gharatmayuresh15@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi Michael, > > > > > > > > > > > > > > > > > > > > > > I have updated the migration > section > > of > > > > the > > > > > > KIP. > > > > > > > Can > > > > > > > > > you > > > > > > > > > > please > > > > > > > > > > > take a look? > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Mayuresh > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 18, 2016 at 9:07 > AM, > > > Mayuresh > > > > > > Gharat < > > > > > > > > > > > gharatmayuresh15@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Michael, > > > > > > > > > > > > > > > > > > > > > > > > That whilst sending > tombstone and > > non > > > > > null > > > > > > value, > > > > > > > > the > > > > > > > > > > consumer > > > > > > > > > > > can expect > > > > > > > > > > > > only to receive the non-nul= l > > message > > > > only > > > > > > in step > > > > > > > > > (3) is > > > > > > > > > > this > > > > > > > > > > > correct? > > > > > > > > > > > > ---> I do agree with you > here. > > > > > > > > > > > > > > > > > > > > > > > > Becket, Ismael : can you gu= ys > > review > > > > the > > > > > > > migration > > > > > > > > > plan > > > > > > > > > > listed > > > > > > > > > > > above using > > > > > > > > > > > > magic byte? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Mayuresh > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 18, 2016 at 8:5= 8 > AM, > > > > Michael > > > > > > Pearce < > > > > > > > > > > > Michael.Pearce@ig.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > >> Many thanks for this > Mayuresh. I > > > don't > > > > > > have any > > > > > > > > > > objections. > > > > > > > > > > > >> > > > > > > > > > > > >> I assume we should state: > > > > > > > > > > > >> > > > > > > > > > > > >> That whilst sending > tombstone and > > > non > > > > > null > > > > > > > value, > > > > > > > > > the > > > > > > > > > > consumer > > > > > > > > > > > can expect > > > > > > > > > > > >> only to receive the non-nu= ll > > message > > > > > only > > > > > > in > > > > > > > step > > > > > > > > > (3) is > > > > > > > > > > this > > > > > > > > > > > correct? > > > > > > > > > > > >> > > > > > > > > > > > >> Cheers > > > > > > > > > > > >> Mike > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Sent using OWA for iPhone > > > > > > > > > > > >> > ______________________________ > > > > > __________ > > > > > > > > > > > >> From: Mayuresh Gharat < > > > > > > > gharatmayuresh15@gmail.com > > > > > > > > > > > > > > > > > > > > >> Sent: Thursday, November > 17, 2016 > > > > > 5:18:41 > > > > > > PM > > > > > > > > > > > >> To: dev@kafka.apache.org > > > > > > > > > > > >> Subject: Re: [DISCUSS] > KIP-87 - > > Add > > > > > > Compaction > > > > > > > > > Tombstone > > > > > > > > > > Flag > > > > > > > > > > > >> > > > > > > > > > > > >> Hi Ismael, > > > > > > > > > > > >> > > > > > > > > > > > >> Thanks for the explanation= . > > > > > > > > > > > >> Specially I like this part > where > > in > > > > you > > > > > > > mentioned > > > > > > > > > we can > > > > > > > > > > get > > > > > > > > > > > rid of the > > > > > > > > > > > >> older null value support > for log > > > > > > compaction > > > > > > > later > > > > > > > > > on, > > > > > > > > > > here : > > > > > > > > > > > >> We can't change semantics > of the > > > > message > > > > > > format > > > > > > > > > without > > > > > > > > > > having > > > > > > > > > > > a long > > > > > > > > > > > >> transition period. And we > can't > > rely > > > > > > > > > > > >> on people reading > documentation or > > > > > acting > > > > > > on a > > > > > > > > > warning for > > > > > > > > > > > something so > > > > > > > > > > > >> fundamental. As such, my > take is > > > that > > > > we > > > > > > need to > > > > > > > > > bump the > > > > > > > > > > magic > > > > > > > > > > > byte. The > > > > > > > > > > > >> good news is > > > > > > > > > > > >> that we don't have to > support all > > > > > versions > > > > > > > > forever. > > > > > > > > > We > > > > > > > > > > have > > > > > > > > > > > said that we > > > > > > > > > > > >> will support direct > upgrades for 2 > > > > > years. > > > > > > That > > > > > > > > > means that > > > > > > > > > > > message format > > > > > > > > > > > >> version n could, in theory= , > be > > > > removed 2 > > > > > > years > > > > > > > > > after the > > > > > > > > > > it's > > > > > > > > > > > introduced. > > > > > > > > > > > >> > > > > > > > > > > > >> Just a heads up, I would > like to > > > > mention > > > > > > that > > > > > > > even > > > > > > > > > without > > > > > > > > > > > bumping magic > > > > > > > > > > > >> byte, we will *NOT* loose > zero > > copy > > > as > > > > > in > > > > > > the > > > > > > > > > client(x+1) > > > > > > > > > > in my > > > > > > > > > > > >> explanation > > > > > > > > > > > >> above will convert > internally a > > null > > > > > > value to > > > > > > > > have a > > > > > > > > > > tombstone > > > > > > > > > > > bit set and > > > > > > > > > > > >> a tombstone bit set to hav= e > a null > > > > value > > > > > > > > > automatically > > > > > > > > > > > internally and by > > > > > > > > > > > >> the time we move to versio= n > (x+2), > > > the > > > > > > clients > > > > > > > > > would have > > > > > > > > > > > upgraded. > > > > > > > > > > > >> Obviously if we support a > request > > > from > > > > > > > > consumer(x), > > > > > > > > > we > > > > > > > > > > will > > > > > > > > > > > loose zero > > > > > > > > > > > >> copy > > > > > > > > > > > >> but that is the same case > with > > magic > > > > > byte. > > > > > > > > > > > >> > > > > > > > > > > > >> But if magic byte bump > makes life > > > > easier > > > > > > for > > > > > > > > > transition > > > > > > > > > > for the > > > > > > > > > > > above > > > > > > > > > > > >> reasons that you explained= , > I am > > OK > > > > with > > > > > > it > > > > > > > since > > > > > > > > > we are > > > > > > > > > > going > > > > > > > > > > > to meet the > > > > > > > > > > > >> end goal down the road :) > > > > > > > > > > > >> > > > > > > > > > > > >> On a side note can we > update the > > doc > > > > > here > > > > > > on > > > > > > > magic > > > > > > > > > byte > > > > > > > > > > to say > > > > > > > > > > > that "*it > > > > > > > > > > > >> should be bumped whenever > the > > > message > > > > > > format is > > > > > > > > > changed > > > > > > > > > > or the > > > > > > > > > > > >> interpretation of message > format > > > > (usage > > > > > > of the > > > > > > > > > reserved > > > > > > > > > > bits as > > > > > > > > > > > well) is > > > > > > > > > > > >> changed*". > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Hi Michael, > > > > > > > > > > > >> > > > > > > > > > > > >> Here is the update plan > that we > > > > > discussed > > > > > > > offline > > > > > > > > > > yesterday : > > > > > > > > > > > >> > > > > > > > > > > > >> Currently the magic-byte > which > > > > > > corresponds to > > > > > > > the > > > > > > > > > > > "message.format.version" > > > > > > > > > > > >> is set to 1. > > > > > > > > > > > >> > > > > > > > > > > > >> 1) On broker it will be se= t > to 1 > > > > > > initially. > > > > > > > > > > > >> > > > > > > > > > > > >> 2) When a producer client > sends a > > > > > message > > > > > > with > > > > > > > > > magic-byte > > > > > > > > > > =3D 2, > > > > > > > > > > > since the > > > > > > > > > > > >> broker is on magic-byte = =3D > 1, we > > will > > > > > down > > > > > > > convert > > > > > > > > > it, > > > > > > > > > > which > > > > > > > > > > > means if the > > > > > > > > > > > >> tombstone bit is set, the > value > > will > > > > be > > > > > > set to > > > > > > > > > null. A > > > > > > > > > > consumer > > > > > > > > > > > >> understanding magic-byte = =3D > 1, will > > > > still > > > > > > work > > > > > > > with > > > > > > > > > this. A > > > > > > > > > > > consumer > > > > > > > > > > > >> working > > > > > > > > > > > >> with magic-byte =3D2 will > also be > > able > > > > to > > > > > > > understand > > > > > > > > > this, > > > > > > > > > > since > > > > > > > > > > > it > > > > > > > > > > > >> understands the tombstone. > > > > > > > > > > > >> Now there is still the > question of > > > > > > supporting a > > > > > > > > > > non-tombstone > > > > > > > > > > > and null > > > > > > > > > > > >> value from producer client > with > > > > > > magic-byte =3D 2.* > > > > > > > > (I > > > > > > > > > am > > > > > > > > > > not sure > > > > > > > > > > > if we > > > > > > > > > > > >> should support this. > Ismael/Becket > > > can > > > > > > comment > > > > > > > > > here)* > > > > > > > > > > > >> > > > > > > > > > > > >> 3) When almost all the > clients > > have > > > > > > upgraded, > > > > > > > the > > > > > > > > > > > message.format.version > > > > > > > > > > > >> on > > > > > > > > > > > >> the broker can be changed > to 2, > > > where > > > > in > > > > > > the > > > > > > > down > > > > > > > > > > conversion in > > > > > > > > > > > the above > > > > > > > > > > > >> step will not happen. If a= t > this > > > point > > > > > we > > > > > > get a > > > > > > > > > consumer > > > > > > > > > > > request from a > > > > > > > > > > > >> older consumer, we might > have to > > > down > > > > > > convert > > > > > > > > where > > > > > > > > > in we > > > > > > > > > > loose > > > > > > > > > > > zero copy, > > > > > > > > > > > >> but these cases should be > rare. > > > > > > > > > > > >> > > > > > > > > > > > >> Becket can you review this > plan > > and > > > > add > > > > > > more > > > > > > > > > details if I > > > > > > > > > > have > > > > > > > > > > > >> missed/wronged something, > before > > we > > > > put > > > > > > it on > > > > > > > KIP. > > > > > > > > > > > >> > > > > > > > > > > > >> Thanks, > > > > > > > > > > > >> > > > > > > > > > > > >> Mayuresh > > > > > > > > > > > >> > > > > > > > > > > > >> On Wed, Nov 16, 2016 at > 11:07 PM, > > > > > Michael > > > > > > > Pearce < > > > > > > > > > > > Michael.Pearce@ig.com> > > > > > > > > > > > >> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> > Thanks guys, for > discussing this > > > > > > offline and > > > > > > > > > getting > > > > > > > > > > some > > > > > > > > > > > consensus. > > > > > > > > > > > >> > > > > > > > > > > > > >> > So its clear for myself > and > > others > > > > > what > > > > > > is > > > > > > > > > proposed now > > > > > > > > > > (i > > > > > > > > > > > think i > > > > > > > > > > > >> > understand, but want to > make > > sure) > > > > > > > > > > > >> > > > > > > > > > > > > >> > Could i ask either > directly > > update > > > > the > > > > > > kip to > > > > > > > > > detail the > > > > > > > > > > > migration > > > > > > > > > > > >> > strategy, or (re-)state > your > > > offline > > > > > > discussed > > > > > > > > and > > > > > > > > > > agreed > > > > > > > > > > > migration > > > > > > > > > > > >> > strategy based on a magi= c > byte > > is > > > in > > > > > > this > > > > > > > > thread. > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > The main original driver > for the > > > KIP > > > > > > was to > > > > > > > > > support > > > > > > > > > > > compaction where > > > > > > > > > > > >> value > > > > > > > > > > > >> > isn't null, based off th= e > > > > discussions > > > > > on > > > > > > > KIP-82 > > > > > > > > > thread. > > > > > > > > > > > >> > > > > > > > > > > > > >> > We should be able to > support > > > > > > non-tombstone + > > > > > > > > null > > > > > > > > > value > > > > > > > > > > by the > > > > > > > > > > > >> completion > > > > > > > > > > > >> > of the KIP, as we noted > when > > > > > discussing > > > > > > this > > > > > > > > kip, > > > > > > > > > having > > > > > > > > > > > logic based on > > > > > > > > > > > >> a > > > > > > > > > > > >> > null value isn't very > clean and > > > also > > > > > > separates > > > > > > > > the > > > > > > > > > > concerns. > > > > > > > > > > > >> > > > > > > > > > > > > >> > As discussed already > though we > > can > > > > > > split this > > > > > > > > into > > > > > > > > > > KIP-87a > > > > > > > > > > > and KIP-87b > > > > > > > > > > > >> > > > > > > > > > > > > >> > Where we look to deliver > KIP-87a > > > on > > > > a > > > > > > > compacted > > > > > > > > > topic > > > > > > > > > > (to > > > > > > > > > > > address the > > > > > > > > > > > >> > immediate issues) > > > > > > > > > > > >> > * tombstone + null value > > > > > > > > > > > >> > * tombstone + non-null > value > > > > > > > > > > > >> > * non-tombstone + > non-null value > > > > > > > > > > > >> > > > > > > > > > > > > >> > Then we can discuss once > KIP-87a > > > is > > > > > > completed > > > > > > > > > options > > > > > > > > > > later > > > > > > > > > > > and how we > > > > > > > > > > > >> > support the second part > KIP-87b > > to > > > > > > deliver: > > > > > > > > > > > >> > * non-tombstone + null > value > > > > > > > > > > > >> > > > > > > > > > > > > >> > Cheers > > > > > > > > > > > >> > Mike > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > ______________________________ > > > > > > __________ > > > > > > > > > > > >> > From: Becket Qin < > > > > > becket.qin@gmail.com> > > > > > > > > > > > >> > Sent: Thursday, November > 17, > > 2016 > > > > 1:43 > > > > > > AM > > > > > > > > > > > >> > To: dev@kafka.apache.org > > > > > > > > > > > >> > Subject: Re: [DISCUSS] > KIP-87 - > > > Add > > > > > > Compaction > > > > > > > > > > Tombstone Flag > > > > > > > > > > > >> > > > > > > > > > > > > >> > Renu, Mayuresh and I had > an > > > offline > > > > > > > discussion, > > > > > > > > > and > > > > > > > > > > following > > > > > > > > > > > is a brief > > > > > > > > > > > >> > summary. > > > > > > > > > > > >> > > > > > > > > > > > > >> > 1. We agreed that not > bumping up > > > > magic > > > > > > value > > > > > > > may > > > > > > > > > result > > > > > > > > > > in > > > > > > > > > > > losing zero > > > > > > > > > > > >> copy > > > > > > > > > > > >> > during migration. > > > > > > > > > > > >> > 2. Given that bumping up > magic > > > value > > > > > is > > > > > > almost > > > > > > > > > free and > > > > > > > > > > has > > > > > > > > > > > benefit of > > > > > > > > > > > >> > avoiding potential > performance > > > > issue. > > > > > > It is > > > > > > > > > probably > > > > > > > > > > worth > > > > > > > > > > > doing. > > > > > > > > > > > >> > > > > > > > > > > > > >> > One issue we still need > to think > > > > about > > > > > > is > > > > > > > > whether > > > > > > > > > we > > > > > > > > > > want to > > > > > > > > > > > support a > > > > > > > > > > > >> > non-tombstone message > with null > > > > value. > > > > > > > > > > > >> > Currently it is not > supported by > > > > > Kafka. > > > > > > If we > > > > > > > > > allow a > > > > > > > > > > > non-tombstone null > > > > > > > > > > > >> > value message to exist > after > > > KIP-87. > > > > > The > > > > > > > problem > > > > > > > > > is > > > > > > > > > > that such > > > > > > > > > > > message > > > > > > > > > > > >> will > > > > > > > > > > > >> > not be supported by the > > consumers > > > > > prior > > > > > > to > > > > > > > > KIP-87. > > > > > > > > > > Because a > > > > > > > > > > > null value > > > > > > > > > > > >> > will always be > interpreted to a > > > > > > tombstone. > > > > > > > > > > > >> > > > > > > > > > > > > >> > One option is that we > keep the > > > > current > > > > > > way, > > > > > > > i.e. > > > > > > > > > do not > > > > > > > > > > > support such > > > > > > > > > > > >> > message. It would be goo= d > to > > know > > > if > > > > > > there is > > > > > > > a > > > > > > > > > > concrete use > > > > > > > > > > > case for > > > > > > > > > > > >> such > > > > > > > > > > > >> > message. If there is not= , > we can > > > > > > probably just > > > > > > > > not > > > > > > > > > > support it. > > > > > > > > > > > >> > > > > > > > > > > > > >> > Thanks, > > > > > > > > > > > >> > > > > > > > > > > > > >> > JIangjie (Becket) Qin > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > On Wed, Nov 16, 2016 at > 1:28 PM, > > > > > > Mayuresh > > > > > > > > Gharat < > > > > > > > > > > > >> > > gharatmayuresh15@gmail.com > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > >> > > Hi Ismael, > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > This is something I ca= n > think > > of > > > > for > > > > > > > migration > > > > > > > > > plan: > > > > > > > > > > > >> > > So the migration plan > can look > > > > > > something > > > > > > > like > > > > > > > > > this, > > > > > > > > > > with up > > > > > > > > > > > >> conversion : > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > 1) Currently lets say > we have > > > > Broker > > > > > > at > > > > > > > > version > > > > > > > > > x. > > > > > > > > > > > >> > > 2) Currently we have > clients > > at > > > > > > version x. > > > > > > > > > > > >> > > 3) a) We move the > version to > > > > > > Broker(x+1) : > > > > > > > > > supports > > > > > > > > > > both > > > > > > > > > > > tombstone and > > > > > > > > > > > >> > null > > > > > > > > > > > >> > > for log compaction. > > > > > > > > > > > >> > > b) We upgrade the > client > > to > > > > > > version > > > > > > > > > client(x+1) : > > > > > > > > > > if in > > > > > > > > > > > the > > > > > > > > > > > >> producer > > > > > > > > > > > >> > > client(x+1) the value > is set > > to > > > > > null, > > > > > > we > > > > > > > will > > > > > > > > > > automatically > > > > > > > > > > > set the > > > > > > > > > > > >> > > Tombstone bit > internally. If > > the > > > > > > producer > > > > > > > > > client(x+1) > > > > > > > > > > sets > > > > > > > > > > > the > > > > > > > > > > > >> tombstone > > > > > > > > > > > >> > > itself, well and good. > For > > > > producer > > > > > > > client(x), > > > > > > > > > the > > > > > > > > > > broker > > > > > > > > > > > will up > > > > > > > > > > > >> convert > > > > > > > > > > > >> > > to have the tombstone > bit. > > > > > > Broker(x+1) is > > > > > > > > > supporting > > > > > > > > > > both. > > > > > > > > > > > Consumer > > > > > > > > > > > >> > > client(x+1) will be > aware of > > > this > > > > > and > > > > > > should > > > > > > > > be > > > > > > > > > able > > > > > > > > > > to > > > > > > > > > > > handle this. > > > > > > > > > > > >> For > > > > > > > > > > > >> > > consumer client(x) we > will > > down > > > > > > convert the > > > > > > > > > message > > > > > > > > > > on the > > > > > > > > > > > broker > > > > > > > > > > > >> side. > > > > > > > > > > > >> > > c) At this point w= e > will > > > have > > > > to > > > > > > > specify a > > > > > > > > > > warning or > > > > > > > > > > > clearly > > > > > > > > > > > >> specify > > > > > > > > > > > >> > > in docs that this > behavior is > > > > about > > > > > > to be > > > > > > > > > changed for > > > > > > > > > > log > > > > > > > > > > > compaction. > > > > > > > > > > > >> > > 4) a) In next release > of the > > > > > > Broker(x+2), we > > > > > > > > > say that > > > > > > > > > > only > > > > > > > > > > > Tombstone > > > > > > > > > > > >> is > > > > > > > > > > > >> > > used for log compactio= n > on the > > > > > Broker > > > > > > side. > > > > > > > > > > Clients(x+1) > > > > > > > > > > > still is > > > > > > > > > > > >> > > supported. > > > > > > > > > > > >> > > b) We upgrade the > client > > to > > > > > > version > > > > > > > > > client(x+2) : > > > > > > > > > > if > > > > > > > > > > > value is set > > > > > > > > > > > >> to > > > > > > > > > > > >> > > null, tombstone will > not be > > set > > > > > > > automatically. > > > > > > > > > The > > > > > > > > > > client > > > > > > > > > > > will have to > > > > > > > > > > > >> > call > > > > > > > > > > > >> > > setTombstone() to > actually set > > > the > > > > > > > tombstone. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > We should compare this > > migration > > > > > plan > > > > > > with > > > > > > > the > > > > > > > > > > migration > > > > > > > > > > > plan for > > > > > > > > > > > >> magic > > > > > > > > > > > >> > > byte bump and do > whatever > > looks > > > > > good. > > > > > > > > > > > >> > > I am just worried that > if we > > go > > > > down > > > > > > magic > > > > > > > > byte > > > > > > > > > route, > > > > > > > > > > > unless I am > > > > > > > > > > > >> > missing > > > > > > > > > > > >> > > something, it sounds > like > > kafka > > > > will > > > > > > be > > > > > > > stuck > > > > > > > > > with > > > > > > > > > > > supporting both > > > > > > > > > > > >> null > > > > > > > > > > > >> > > value and tombstone bi= t > for > > log > > > > > > compaction > > > > > > > for > > > > > > > > > life > > > > > > > > > > long, > > > > > > > > > > > which does > > > > > > > > > > > >> not > > > > > > > > > > > >> > > look like a good end > state. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Thanks, > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Mayuresh > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > On Wed, Nov 16, 2016 a= t > 9:32 > > AM, > > > > > > Mayuresh > > > > > > > > > Gharat < > > > > > > > > > > > >> > > > gharatmayuresh15@gmail.com > > > > > > > > > > > >> > > > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > Hi Ismael, > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > That's a very good > point > > > which I > > > > > > might > > > > > > > have > > > > > > > > > not > > > > > > > > > > > considered earlier. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Here is a plan that = I > can > > > think > > > > > of: > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Stage 1) The broker > from now > > > on, > > > > > up > > > > > > > converts > > > > > > > > > the > > > > > > > > > > message > > > > > > > > > > > to have the > > > > > > > > > > > >> > > > tombstone marker. Th= e > log > > > > > compaction > > > > > > > thread > > > > > > > > > does log > > > > > > > > > > > compaction > > > > > > > > > > > >> based > > > > > > > > > > > >> > on > > > > > > > > > > > >> > > > both null and > tombstone > > > marker. > > > > > > This is > > > > > > > our > > > > > > > > > > transition > > > > > > > > > > > period. > > > > > > > > > > > >> > > > Stage 2) The next > release we > > > > only > > > > > > say that > > > > > > > > log > > > > > > > > > > compaction > > > > > > > > > > > is based > > > > > > > > > > > >> on > > > > > > > > > > > >> > > > tombstone marker. > (Open > > source > > > > > > kafka makes > > > > > > > > > this as a > > > > > > > > > > > policy). By > > > > > > > > > > > >> this > > > > > > > > > > > >> > > time, > > > > > > > > > > > >> > > > the organization > which is > > > moving > > > > > to > > > > > > this > > > > > > > > > release > > > > > > > > > > will be > > > > > > > > > > > sure that > > > > > > > > > > > >> they > > > > > > > > > > > >> > > > have gone through th= e > entire > > > > > > transition > > > > > > > > > period. > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > My only goal of doin= g > this > > is > > > > that > > > > > > Kafka > > > > > > > > > clearly > > > > > > > > > > > specifies the end > > > > > > > > > > > >> > state > > > > > > > > > > > >> > > > about what log > compaction > > > means > > > > > (is > > > > > > it > > > > > > > null > > > > > > > > > value > > > > > > > > > > or a > > > > > > > > > > > tombstone > > > > > > > > > > > >> > marker, > > > > > > > > > > > >> > > > but not both). > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > What do you think? > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Thanks, > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > Mayuresh > > > > > > > > > > > >> > > > . > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > On Wed, Nov 16, 2016 > at 9:17 > > > AM, > > > > > > Ismael > > > > > > > > Juma < > > > > > > > > > > > ismael@juma.me.uk> > > > > > > > > > > > >> > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > >> One comment below. > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> On Wed, Nov 16, 201= 6 > at > > 5:08 > > > > PM, > > > > > > Mayuresh > > > > > > > > > Gharat < > > > > > > > > > > > >> > > >> > gharatmayuresh15@gmail.com > > > > > > > > > > > >> > > >> > wrote: > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > - If we don't > bump up > > > the > > > > > > magic > > > > > > > byte, > > > > > > > > > on the > > > > > > > > > > broker > > > > > > > > > > > side, the > > > > > > > > > > > >> > > broker > > > > > > > > > > > >> > > >> > will always > have to > > look > > > > at > > > > > > both > > > > > > > > > tombstone > > > > > > > > > > bit and > > > > > > > > > > > the value > > > > > > > > > > > >> when > > > > > > > > > > > >> > > do > > > > > > > > > > > >> > > >> the > > > > > > > > > > > >> > > >> > compaction. > Assuming > > we > > > do > > > > > > not bump > > > > > > > up > > > > > > > > > the > > > > > > > > > > magic > > > > > > > > > > > byte, > > > > > > > > > > > >> > > >> > imagine the > broker > > sees > > > a > > > > > > message > > > > > > > > which > > > > > > > > > does > > > > > > > > > > not > > > > > > > > > > > have a > > > > > > > > > > > >> tombstone > > > > > > > > > > > >> > > bit > > > > > > > > > > > >> > > >> > set. The broke= r > does > > not > > > > > know > > > > > > when > > > > > > > the > > > > > > > > > > message was > > > > > > > > > > > produced > > > > > > > > > > > >> (i.e. > > > > > > > > > > > >> > > >> > whether > > > > > > > > > > > >> > > >> > the message ha= s > been > > up > > > > > > converted or > > > > > > > > > not), it > > > > > > > > > > has > > > > > > > > > > > to take a > > > > > > > > > > > >> > further > > > > > > > > > > > >> > > >> > look at > > > > > > > > > > > >> > > >> > the value to > see if it > > > is > > > > > > null or > > > > > > > not > > > > > > > > in > > > > > > > > > > order to > > > > > > > > > > > determine > > > > > > > > > > > >> if it > > > > > > > > > > > >> > > is > > > > > > > > > > > >> > > >> a > > > > > > > > > > > >> > > >> > tombstone. The > same > > > logic > > > > > has > > > > > > to be > > > > > > > > put > > > > > > > > > on the > > > > > > > > > > > consumer as > > > > > > > > > > > >> well > > > > > > > > > > > >> > > >> because > > > > > > > > > > > >> > > >> > the > > > > > > > > > > > >> > > >> > consumer does > not know > > > if > > > > > the > > > > > > > message > > > > > > > > > has > > > > > > > > > > been up > > > > > > > > > > > converted or > > > > > > > > > > > >> > not. > > > > > > > > > > > >> > > >> > - If we > upconvert > > > while > > > > > > > appending, > > > > > > > > > this is > > > > > > > > > > not > > > > > > > > > > > the case, > > > > > > > > > > > >> > right? > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> If I understand you > > > correctly, > > > > > > this is > > > > > > > not > > > > > > > > > > sufficient > > > > > > > > > > > because the > > > > > > > > > > > >> log > > > > > > > > > > > >> > > may > > > > > > > > > > > >> > > >> have messages > appended > > before > > > > it > > > > > > was > > > > > > > > > upgraded to > > > > > > > > > > include > > > > > > > > > > > KIP-87. > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > >> Ismael > > > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > -- > > > > > > > > > > > >> > > > -Regards, > > > > > > > > > > > >> > > > Mayuresh R. Gharat > > > > > > > > > > > >> > > > (862) 250-7125 > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > -- > > > > > > > > > > > >> > > -Regards, > > > > > > > > > > > >> > > Mayuresh R. Gharat > > > > > > > > > > > >> > > (862) 250-7125 > > > > > > > > > > > >> > > > > > > > > > > > > > >> > The information containe= d > in > > this > > > > > email > > > > > > is > > > > > > > > > strictly > > > > > > > > > > > confidential and for > > > > > > > > > > > >> > the use of the addressee > only, > > > > unless > > > > > > > otherwise > > > > > > > > > > indicated. If > > > > > > > > > > > you are > > > > > > > > > > > >> not > > > > > > > > > > > >> > the intended recipient, > please > > do > > > > not > > > > > > read, > > > > > > > > copy, > > > > > > > > > use or > > > > > > > > > > > disclose to > > > > > > > > > > > >> others > > > > > > > > > > > >> > this message or any > attachment. > > > > Please > > > > > > also > > > > > > > > > notify the > > > > > > > > > > sender > > > > > > > > > > > by > > > > > > > > > > > >> replying > > > > > > > > > > > >> > to this email or by > telephone > > > > (+44(020 > > > > > > 7896 > > > > > > > > 0011) > > > > > > > > > and > > > > > > > > > > then > > > > > > > > > > > delete the > > > > > > > > > > > >> email > > > > > > > > > > > >> > and any copies of it. > Opinions, > > > > > > conclusion > > > > > > > (etc) > > > > > > > > > that > > > > > > > > > > do not > > > > > > > > > > > relate to > > > > > > > > > > > >> the > > > > > > > > > > > >> > official business of thi= s > > company > > > > > shall > > > > > > be > > > > > > > > > understood as > > > > > > > > > > > neither given > > > > > > > > > > > >> nor > > > > > > > > > > > >> > endorsed by it. IG is a > trading > > > name > > > > > of > > > > > > IG > > > > > > > > Markets > > > > > > > > > > Limited (a > > > > > > > > > > > company > > > > > > > > > > > >> > registered in England an= d > Wales, > > > > > company > > > > > > > number > > > > > > > > > > 04008957) and > > > > > > > > > > > IG Index > > > > > > > > > > > >> > Limited (a company > registered in > > > > > > England and > > > > > > > > > Wales, > > > > > > > > > > company > > > > > > > > > > > number > > > > > > > > > > > >> > 01190902). Registered > address at > > > > > Cannon > > > > > > Bridge > > > > > > > > > House, 25 > > > > > > > > > > > Dowgate Hill, > > > > > > > > > > > >> > London EC4R 2YA. Both IG > Markets > > > > > Limited > > > > > > > > (register > > > > > > > > > > number > > > > > > > > > > > 195355) and IG > > > > > > > > > > > >> > Index Limited (register > number > > > > 114059) > > > > > > are > > > > > > > > > authorised > > > > > > > > > > and > > > > > > > > > > > regulated by > > > > > > > > > > > >> the > > > > > > > > > > > >> > Financial Conduct > Authority. > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> -- > > > > > > > > > > > >> -Regards, > > > > > > > > > > > >> Mayuresh R. Gharat > > > > > > > > > > > >> (862) 250-7125 > > > > > > > > > > > >> The information contained > in this > > > > email > > > > > is > > > > > > > > strictly > > > > > > > > > > > confidential and for > > > > > > > > > > > >> the use of the addressee > only, > > > unless > > > > > > otherwise > > > > > > > > > > indicated. If > > > > > > > > > > > you are not > > > > > > > > > > > >> the intended recipient, > please do > > > not > > > > > > read, > > > > > > > copy, > > > > > > > > > use or > > > > > > > > > > > disclose to others > > > > > > > > > > > >> this message or any > attachment. > > > Please > > > > > > also > > > > > > > notify > > > > > > > > > the > > > > > > > > > > sender > > > > > > > > > > > by replying > > > > > > > > > > > >> to this email or by > telephone > > > (+44(020 > > > > > > 7896 > > > > > > > 0011) > > > > > > > > > and then > > > > > > > > > > > delete the email > > > > > > > > > > > >> and any copies of it. > Opinions, > > > > > > conclusion (etc) > > > > > > > > > that do > > > > > > > > > > not > > > > > > > > > > > relate to the > > > > > > > > > > > >> official business of this > company > > > > shall > > > > > be > > > > > > > > > understood as > > > > > > > > > > > neither given nor > > > > > > > > > > > >> endorsed by it. IG is a > trading > > name > > > > of > > > > > IG > > > > > > > Markets > > > > > > > > > > Limited (a > > > > > > > > > > > company > > > > > > > > > > > >> registered in England and > Wales, > > > > company > > > > > > number > > > > > > > > > 04008957) > > > > > > > > > > and > > > > > > > > > > > IG Index > > > > > > > > > > > >> Limited (a company > registered in > > > > England > > > > > > and > > > > > > > > Wales, > > > > > > > > > > company > > > > > > > > > > > number > > > > > > > > > > > >> 01190902). Registered > address at > > > > Cannon > > > > > > Bridge > > > > > > > > > House, 25 > > > > > > > > > > > Dowgate Hill, > > > > > > > > > > > >> London EC4R 2YA. Both IG > Markets > > > > Limited > > > > > > > (register > > > > > > > > > number > > > > > > > > > > > 195355) and IG > > > > > > > > > > > >> Index Limited (register > number > > > 114059) > > > > > are > > > > > > > > > authorised and > > > > > > > > > > > regulated by the > > > > > > > > > > > >> Financial Conduct Authorit= y. > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > -Regards, > > > > > > > > > > > > Mayuresh R. Gharat > > > > > > > > > > > > (862) 250-7125 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > -Regards, > > > > > > > > > > > Mayuresh R. Gharat > > > > > > > > > > > (862) 250-7125 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this > email > > is > > > > > > strictly > > > > > > > > > confidential > > > > > > > > > > and > > > > > > > > > > > for the use of the addressee only, > unless > > > > otherwise > > > > > > > > indicated. > > > > > > > > > If > > > > > > > > > > you are > > > > > > > > > > > not the intended recipient, please do > not > > read, > > > > > > copy, use > > > > > > > or > > > > > > > > > > disclose to > > > > > > > > > > > others this message or any attachment= . > Please > > > > also > > > > > > notify > > > > > > > the > > > > > > > > > sender > > > > > > > > > > by > > > > > > > > > > > replying to this email or by telephon= e > > (+44(020 > > > > > 7896 > > > > > > 0011) > > > > > > > > and > > > > > > > > > then > > > > > > > > > > delete > > > > > > > > > > > the email and any copies of it. > Opinions, > > > > > conclusion > > > > > > (etc) > > > > > > > > > that do > > > > > > > > > > not > > > > > > > > > > > relate to the official business of th= is > > company > > > > > > shall be > > > > > > > > > understood > > > > > > > > > > as > > > > > > > > > > > neither given nor endorsed by it. IG > is a > > > trading > > > > > > name of > > > > > > > IG > > > > > > > > > Markets > > > > > > > > > > > Limited (a company registered in > England and > > > > Wales, > > > > > > company > > > > > > > > > number > > > > > > > > > > > 04008957) and IG Index Limited (a > company > > > > > registered > > > > > > in > > > > > > > > > England and > > > > > > > > > > Wales, > > > > > > > > > > > company number 01190902). Registered > address > > at > > > > > > Cannon > > > > > > > Bridge > > > > > > > > > House, > > > > > > > > > > 25 > > > > > > > > > > > Dowgate Hill, London EC4R 2YA. Both I= G > > Markets > > > > > > Limited > > > > > > > > > (register > > > > > > > > > > number > > > > > > > > > > > 195355) and IG Index Limited (registe= r > number > > > > > > 114059) are > > > > > > > > > authorised > > > > > > > > > > and > > > > > > > > > > > regulated by the Financial Conduct > Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is > strictly > > > > > > confidential > > > > > > > > and > > > > > > > > > for > > > > > > > > > > the use of the addressee only, unless > otherwise > > > > > indicated. > > > > > > If you > > > > > > > > > are not > > > > > > > > > > the intended recipient, please do not read, > copy, > > use > > > > or > > > > > > disclose > > > > > > > > to > > > > > > > > > others > > > > > > > > > > this message or any attachment. Please also > notify > > > the > > > > > > sender by > > > > > > > > > replying > > > > > > > > > > to this email or by telephone (+44(020 7896 > 0011 > > <020%207896%200011>) and > > > > > then > > > > > > delete > > > > > > > > > the email > > > > > > > > > > and any copies of it. Opinions, conclusion > (etc) > > that > > > > do > > > > > > not > > > > > > > relate > > > > > > > > > to the > > > > > > > > > > official business of this company shall be > > understood > > > > as > > > > > > neither > > > > > > > > > given nor > > > > > > > > > > endorsed by it. IG is a trading name of IG > Markets > > > > > Limited > > > > > > (a > > > > > > > > company > > > > > > > > > > registered in England and Wales, company > number > > > > 04008957) > > > > > > and IG > > > > > > > > > Index > > > > > > > > > > Limited (a company registered in England an= d > Wales, > > > > > company > > > > > > > number > > > > > > > > > > 01190902). Registered address at Cannon > Bridge > > House, > > > > 25 > > > > > > Dowgate > > > > > > > > > Hill, > > > > > > > > > > London EC4R 2YA. Both IG Markets Limited > (register > > > > number > > > > > > 195355) > > > > > > > > > and IG > > > > > > > > > > Index Limited (register number 114059) are > > authorised > > > > and > > > > > > > regulated > > > > > > > > > by the > > > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strict= ly > > > > > confidential > > > > > > and > > > > > > > for > > > > > > > > > the use of the addressee only, unless otherwise > > indicated. > > > If > > > > > > you are > > > > > > > not > > > > > > > > > the intended recipient, please do not read, copy, > use or > > > > > > disclose to > > > > > > > > others > > > > > > > > > this message or any attachment. Please also notif= y > the > > > sender > > > > > by > > > > > > > replying > > > > > > > > > to this email or by telephone (+44(020 7896 0011 > > <020%207896%200011>) and then > > > > > > delete the > > > > > > > > email > > > > > > > > > and any copies of it. Opinions, conclusion (etc) > that do > > > not > > > > > > relate to > > > > > > > > the > > > > > > > > > official business of this company shall be > understood as > > > > > neither > > > > > > given > > > > > > > > nor > > > > > > > > > endorsed by it. IG is a trading name of IG Market= s > > Limited > > > (a > > > > > > company > > > > > > > > > registered in England and Wales, company number > 04008957) > > > and > > > > > IG > > > > > > Index > > > > > > > > > Limited (a company registered in England and Wale= s, > > company > > > > > > number > > > > > > > > > 01190902). Registered address at Cannon Bridge > House, 25 > > > > > Dowgate > > > > > > Hill, > > > > > > > > > London EC4R 2YA. Both IG Markets Limited (registe= r > number > > > > > > 195355) and > > > > > > > IG > > > > > > > > > Index Limited (register number 114059) are > authorised and > > > > > > regulated by > > > > > > > > the > > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > The information contained in this email is strictly > > > > confidential > > > > > > and for > > > > > > > > the use of the addressee only, unless otherwise > indicated. > > If > > > > you > > > > > > are not > > > > > > > > the intended recipient, please do not read, copy, > use or > > > > disclose > > > > > > to > > > > > > > others > > > > > > > > this message or any attachment. Please also notify > the > > sender > > > > by > > > > > > replying > > > > > > > > to this email or by telephone (+44(020 7896 0011 > > <020%207896%200011>) and then > > > > delete > > > > > > the > > > > > > > email > > > > > > > > and any copies of it. Opinions, conclusion (etc) > that do > > not > > > > > > relate to > > > > > > > the > > > > > > > > official business of this company shall be > understood as > > > > neither > > > > > > given > > > > > > > nor > > > > > > > > endorsed by it. IG is a trading name of IG Markets > Limited > > (a > > > > > > company > > > > > > > > registered in England and Wales, company number > 04008957) > > and > > > > IG > > > > > > Index > > > > > > > > Limited (a company registered in England and Wales, > company > > > > > number > > > > > > > > 01190902). Registered address at Cannon Bridge > House, 25 > > > > Dowgate > > > > > > Hill, > > > > > > > > London EC4R 2YA. Both IG Markets Limited (register > number > > > > 195355) > > > > > > and IG > > > > > > > > Index Limited (register number 114059) are > authorised and > > > > > > regulated by > > > > > > > the > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > > > confidential > > > > > and > > > > > > for > > > > > > > the use of the addressee only, unless otherwise > indicated. If > > > you > > > > > > are not > > > > > > > the intended recipient, please do not read, copy, use > or > > > disclose > > > > > to > > > > > > others > > > > > > > this message or any attachment. Please also notify th= e > sender > > > by > > > > > > replying > > > > > > > to this email or by telephone (+44(020 7896 0011 > > <020%207896%200011>) and then > > > delete > > > > > > the email > > > > > > > and any copies of it. Opinions, conclusion (etc) that > do not > > > > relate > > > > > > to the > > > > > > > official business of this company shall be understood > as > > > neither > > > > > > given nor > > > > > > > endorsed by it. IG is a trading name of IG Markets > Limited (a > > > > > company > > > > > > > registered in England and Wales, company number > 04008957) and > > > IG > > > > > > Index > > > > > > > Limited (a company registered in England and Wales, > company > > > > number > > > > > > > 01190902). Registered address at Cannon Bridge House, > 25 > > > Dowgate > > > > > > Hill, > > > > > > > London EC4R 2YA. Both IG Markets Limited (register > number > > > 195355) > > > > > > and IG > > > > > > > Index Limited (register number 114059) are authorised > and > > > > regulated > > > > > > by the > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > confidential > > and > > > > for > > > > > > the use of the addressee only, unless otherwise indicated. > If you > > are > > > > not > > > > > > the intended recipient, please do not read, copy, use or > disclose > > to > > > > > others > > > > > > this message or any attachment. Please also notify the > sender by > > > > replying > > > > > > to this email or by telephone (+44(020 7896 0011 > > <020%207896%200011>) and then delete the > > > > > email > > > > > > and any copies of it. Opinions, conclusion (etc) that do no= t > relate > > > to > > > > > the > > > > > > official business of this company shall be understood as > neither > > > given > > > > > nor > > > > > > endorsed by it. IG is a trading name of IG Markets Limited = (a > > company > > > > > > registered in England and Wales, company number 04008957) > and IG > > > Index > > > > > > Limited (a company registered in England and Wales, company > number > > > > > > 01190902). Registered address at Cannon Bridge House, 25 > Dowgate > > > Hill, > > > > > > London EC4R 2YA. Both IG Markets Limited (register number > 195355) > > and > > > > IG > > > > > > Index Limited (register number 114059) are authorised and > regulated > > > by > > > > > the > > > > > > Financial Conduct Authority. > > > > > > > > > > > The information contained in this email is strictly > confidential and > > > for > > > > > the use of the addressee only, unless otherwise indicated. If > you are > > > not > > > > > the intended recipient, please do not read, copy, use or > disclose to > > > > others > > > > > this message or any attachment. Please also notify the sender > by > > > replying > > > > > to this email or by telephone (+44(020 7896 0011 > <020%207896%200011>) > > and then delete the > > > > email > > > > > and any copies of it. Opinions, conclusion (etc) that do not > relate > > to > > > > the > > > > > official business of this company shall be understood as > neither > > given > > > > nor > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > company > > > > > registered in England and Wales, company number 04008957) and > IG > > Index > > > > > Limited (a company registered in England and Wales, company > number > > > > > 01190902). Registered address at Cannon Bridge House, 25 > Dowgate > > Hill, > > > > > London EC4R 2YA. Both IG Markets Limited (register number > 195355) and > > > IG > > > > > Index Limited (register number 114059) are authorised and > regulated > > by > > > > the > > > > > Financial Conduct Authority. > > > > > > > > > The information contained in this email is strictly confidentia= l > and > > for > > > > the use of the addressee only, unless otherwise indicated. If > you are > > not > > > > the intended recipient, please do not read, copy, use or > disclose to > > > others > > > > this message or any attachment. Please also notify the sender b= y > > replying > > > > to this email or by telephone (+44(020 7896 0011 > <020%207896%200011>) > > and then delete the > > > email > > > > and any copies of it. Opinions, conclusion (etc) that do not > relate to > > > the > > > > official business of this company shall be understood as neithe= r > given > > > nor > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > company > > > > registered in England and Wales, company number 04008957) and I= G > Index > > > > Limited (a company registered in England and Wales, company > number > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgat= e > Hill, > > > > London EC4R 2YA. Both IG Markets Limited (register number > 195355) and > > IG > > > > Index Limited (register number 114059) are authorised and > regulated by > > > the > > > > Financial Conduct Authority. > > > > > > > The information contained in this email is strictly confidential > and for > > > the use of the addressee only, unless otherwise indicated. If you > are not > > > the intended recipient, please do not read, copy, use or disclose > to > > others > > > this message or any attachment. Please also notify the sender by > replying > > > to this email or by telephone (+44(020 7896 0011 > <020%207896%200011>) > > and then delete the email > > > and any copies of it. Opinions, conclusion (etc) that do not > relate to > > the > > > official business of this company shall be understood as neither > given > > nor > > > endorsed by it. IG is a trading name of IG Markets Limited (a > company > > > registered in England and Wales, company number 04008957) and IG > Index > > > Limited (a company registered in England and Wales, company numbe= r > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate > Hill, > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) > and IG > > > Index Limited (register number 114059) are authorised and > regulated by > > the > > > Financial Conduct Authority. > > > > > > > > The information contained in this email is strictly confidential and for > the use of the addressee only, unless otherwise indicated. If you are not > the intended recipient, please do not read, copy, use or disclose to othe= rs > this message or any attachment. Please also notify the sender by replying > to this email or by telephone (+44(020 7896 0011) and then delete the ema= il > and any copies of it. Opinions, conclusion (etc) that do not relate to th= e > official business of this company shall be understood as neither given no= r > endorsed by it. IG is a trading name of IG Markets Limited (a company > registered in England and Wales, company number 04008957) and IG Index > Limited (a company registered in England and Wales, company number > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG > Index Limited (register number 114059) are authorised and regulated by th= e > Financial Conduct Authority. > --047d7bd91e9eed11bf0543cb4e6f--