Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E3020200B53 for ; Tue, 12 Jul 2016 21:55:32 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id E1681160A56; Tue, 12 Jul 2016 19:55:32 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DCD58160A53 for ; Tue, 12 Jul 2016 21:55:31 +0200 (CEST) Received: (qmail 75890 invoked by uid 500); 12 Jul 2016 19:55:26 -0000 Mailing-List: contact dev-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list dev@flink.apache.org Received: (qmail 75877 invoked by uid 99); 12 Jul 2016 19:55:25 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jul 2016 19:55:25 +0000 Received: from mail-it0-f42.google.com (mail-it0-f42.google.com [209.85.214.42]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 842CB1A0156 for ; Tue, 12 Jul 2016 19:55:25 +0000 (UTC) Received: by mail-it0-f42.google.com with SMTP id f6so2788468ith.1 for ; Tue, 12 Jul 2016 12:55:25 -0700 (PDT) X-Gm-Message-State: ALyK8tLXFMRVsqEiqWcRmeECGvDXjGkJ83YyRfbYvvLz6vxE/rX40pel/wRHDr0AZSSiRACqMaI6muL48C78HA== X-Received: by 10.36.29.5 with SMTP id 5mr4693519itj.97.1468353324745; Tue, 12 Jul 2016 12:55:24 -0700 (PDT) MIME-Version: 1.0 References: <87fe3f69-58bb-527c-5111-ce9eb86de7ee@apache.org> In-Reply-To: From: Aljoscha Krettek Date: Tue, 12 Jul 2016 19:55:15 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [DISCUSS] FLIP 1 - Flink Improvement Proposal To: "dev@flink.apache.org" Content-Type: multipart/alternative; boundary=001a1135b19453a785053775a6c1 archived-at: Tue, 12 Jul 2016 19:55:33 -0000 --001a1135b19453a785053775a6c1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I'll make the changes to the list of things that we consider public "API" and also add these to the FLIP Template. Then we should be good to go. I'll convert the design docs I started to FLIPs in the next few days and will also ask other people that have authored design docs to move them to FLIPs. On Fri, 8 Jul 2016 at 17:44 Aljoscha Krettek wrote: > I was planning to ask people to do that once the process is in place, yes= . > > Wouldn't it be "forward porting", though? :-) > > On Fri, 8 Jul 2016 at 16:38 Robert Metzger wrote: > >> What about "backporting" some of the currently ongoing FLIPs to the new >> process? >> In particular those which are not yet fully implemented (Security, Mesos= , >> Key Groups, ...) ? >> >> On Fri, Jul 8, 2016 at 4:21 PM, Aljoscha Krettek >> wrote: >> >> > Of course, this would make the list look like this: >> > >> > All of the following are public interfaces that people build around: >> > - DataStream and DataSet API, including classes related to that, such >> as >> > StreamExecutionEnvironment >> > - Classes marked with the @Public annotation >> > - On-disk binary formats, such as checkpoints/savepoints >> > - User-facing scripts, i.e. bin/flink, Yarn scripts, Mesos scripts >> > - Configuration >> > - Monitoring >> > >> > On Fri, 8 Jul 2016 at 16:00 Ufuk Celebi wrote: >> > >> > > What about the scripts? >> > > >> > > On Fri, Jul 8, 2016 at 2:47 PM, Aljoscha Krettek > > >> > > wrote: >> > > > Thanks a lot, Matthias! I created an actual "FLIP-Template" based = on >> > the >> > > > template you copied over. >> > > > >> > > > I propose to change the list of public interfaces to this: >> > > > >> > > > All of the following are public interfaces that people build aroun= d: >> > > > - DataStream and DataSet API, including classes related to that, >> such >> > as >> > > > StreamExecutionEnvironment >> > > > - Classes marked with the @Public annotation >> > > > - On-disk binary formats, such as checkpoints/savepoints >> > > > - Configuration >> > > > - Monitoring >> > > > >> > > > Cheers, >> > > > Aljoscha >> > > > >> > > > On Fri, 8 Jul 2016 at 13:49 Matthias J. Sax >> wrote: >> > > > >> > > >> Done. >> > > >> >> > > >> I replaced KIP -> FLIP and Kafka -> Flink. Still there are some >> things >> > > >> we need to rework. Please have a look and propose changes. >> > > >> >> > > >> >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Prop= osals >> > > >> >> > > >> Same for FLIP Template >> > > >> >> > > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template >> > > >> >> > > >> >> > > >> -Matthias >> > > >> >> > > >> On 07/08/2016 09:56 AM, Aljoscha Krettek wrote: >> > > >> > Hi, >> > > >> > I got this reply from one of the Kafka committers: >> > > >> > >> > > >> > Thanks for sharing your intention to use a process similar to o= ur >> > KIP >> > > >> > >> > > >> > process. You are more than welcome to copy the structures and >> docs >> > > that >> > > >> we >> > > >> > >> > > >> > have for the KIP process. :) >> > > >> > >> > > >> > Ismael >> > > >> > >> > > >> > >> > > >> > So it seems we're good to go. @Matthias, since you are a Kafka >> > > >> contributor, >> > > >> > could you maybe copy the relevant docs from the Kafka wiki to o= ur >> > > wiki? >> > > >> The >> > > >> > source for this page: >> > > >> > >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Prop= osals >> > > >> > and >> > > >> > the KIP template that is referenced there. If you want, I can >> take >> > it >> > > >> from >> > > >> > there and adapt it to Flink and then let everyone discuss it >> again. >> > > >> > >> > > >> > >> > > >> > Cheers, >> > > >> > >> > > >> > Aljoscha >> > > >> > >> > > >> > On Thu, 7 Jul 2016 at 11:38 Aljoscha Krettek < >> aljoscha@apache.org> >> > > >> wrote: >> > > >> > >> > > >> >> I'll reach out to the Kafka community and ask if it's ok for u= s >> to >> > > >> "steal" >> > > >> >> their process for this. >> > > >> >> >> > > >> >> On Thu, 7 Jul 2016 at 11:36 Aljoscha Krettek < >> aljoscha@apache.org> >> > > >> wrote: >> > > >> >> >> > > >> >>> @Matthias: Yes, this is the reason why I like the KIP process >> and >> > > why I >> > > >> >>> said "The problem with these is that a) the comments on the >> Google >> > > >> Docs are >> > > >> >>> not reflected in Jira and the mailing list. There has been so= me >> > very >> > > >> active >> > > >> >>> discussion on some of the docs that most people would never >> > > notice.". >> > > >> >>> >> > > >> >>> On Thu, 7 Jul 2016 at 11:28 Robert Metzger < >> rmetzger@apache.org> >> > > >> wrote: >> > > >> >>> >> > > >> >>>> I also like the proposal. I think its an issue that Google >> Docs >> > > >> comments >> > > >> >>>> are not reflected within ASF infra. Therefore, I'm +1 on >> > discussing >> > > >> the >> > > >> >>>> proposals on the mailing list. >> > > >> >>>> >> > > >> >>>> I agree that we need to clean up our wiki. >> > > >> >>>> >> > > >> >>>> On Thu, Jul 7, 2016 at 10:58 AM, Matthias J. Sax < >> > mjsax@apache.org >> > > > >> > > >> >>>> wrote: >> > > >> >>>> >> > > >> >>>>> Just to point out one thing about Kafka KIPs and using the >> > project >> > > >> >>>> wiki: >> > > >> >>>>> >> > > >> >>>>> The wiki contains the current state of the proposal, while >> the >> > > >> >>>>> discussion is covered over the dev-mailing list. IMHO, this >> > makes >> > > a >> > > >> lot >> > > >> >>>>> of sense, as people tend to follow the mailing list but not >> wiki >> > > >> >>>>> changes. Furthermore, the mailing list tracks the whole >> > discussion >> > > >> >>>>> history, while the proposal is kept in a clean state and th= us >> > > easy to >> > > >> >>>> read. >> > > >> >>>>> >> > > >> >>>>> -Matthias >> > > >> >>>>> >> > > >> >>>>> >> > > >> >>>>> On 07/06/2016 10:09 PM, Aljoscha Krettek wrote: >> > > >> >>>>>> Jip, that's why I referenced the Kafka process which is >> also in >> > > >> their >> > > >> >>>>> wiki. >> > > >> >>>>>> >> > > >> >>>>>> On Wed, 6 Jul 2016 at 21:01 Stephan Ewen >> > > wrote: >> > > >> >>>>>> >> > > >> >>>>>>> Yes, big +1 >> > > >> >>>>>>> >> > > >> >>>>>>> I had actually talked about the same thing with some >> people as >> > > >> well. >> > > >> >>>>>>> >> > > >> >>>>>>> I am currently sketching a few FLIPs for things, like >> > > improvements >> > > >> >>>> to >> > > >> >>>>> the >> > > >> >>>>>>> Yarn/Mesos/Kubernetes integration >> > > >> >>>>>>> >> > > >> >>>>>>> >> > > >> >>>>>>> One thing we should do here is to actually structure the >> wiki >> > a >> > > bit >> > > >> >>>> to >> > > >> >>>>> make >> > > >> >>>>>>> it easier to find information and proposals. >> > > >> >>>>>>> >> > > >> >>>>>>> >> > > >> >>>>>>> >> > > >> >>>>>>> >> > > >> >>>>>>> On Wed, Jul 6, 2016 at 4:24 PM, Ufuk Celebi < >> uce@apache.org> >> > > >> wrote: >> > > >> >>>>>>> >> > > >> >>>>>>>> Hey Aljoscha, >> > > >> >>>>>>>> >> > > >> >>>>>>>> thanks for this proposal. I've somehow missed it last >> week. I >> > > like >> > > >> >>>> the >> > > >> >>>>>>>> idea very much and agree with your assessment about the >> > > problems >> > > >> >>>> with >> > > >> >>>>>>>> the Google Doc approach. >> > > >> >>>>>>>> >> > > >> >>>>>>>> Regarding the process: I'm also in favour of adopting it >> from >> > > >> >>>> Kafka. I >> > > >> >>>>>>>> would not expect any problems with this, but we can post= a >> > > quick >> > > >> >>>> note >> > > >> >>>>>>>> to their ML. >> > > >> >>>>>>>> >> > > >> >>>>>>>> @Matthias: The name works for me. ;-) >> > > >> >>>>>>>> >> > > >> >>>>>>>> =E2=80=93 Ufuk >> > > >> >>>>>>>> >> > > >> >>>>>>>> On Tue, Jun 28, 2016 at 10:19 PM, Matthias J. Sax < >> > > >> >>>> mjsax@apache.org> >> > > >> >>>>>>>> wrote: >> > > >> >>>>>>>>> FLIP ?? Really? :D >> > > >> >>>>>>>>> >> > > >> >>>>>>>>> http://www.maya.tv/en/character/flip >> > > >> >>>>>>>>> >> > > >> >>>>>>>>> -Matthias >> > > >> >>>>>>>>> >> > > >> >>>>>>>>> >> > > >> >>>>>>>>> On 06/28/2016 06:26 PM, Aljoscha Krettek wrote: >> > > >> >>>>>>>>>> I'm proposing to add a formal process for how we deal >> with >> > > >> >>>> (major) >> > > >> >>>>>>>>>> improvements to Flink and design docs. This has been >> > > mentioned >> > > >> >>>>> several >> > > >> >>>>>>>>>> times recently but we never took any decisive action t= o >> > > actually >> > > >> >>>>>>>> implement >> > > >> >>>>>>>>>> such a process so here we go. >> > > >> >>>>>>>>>> >> > > >> >>>>>>>>>> Right now, we have Jira issues and we sometimes we hav= e >> > > design >> > > >> >>>> docs >> > > >> >>>>>>>> that we >> > > >> >>>>>>>>>> keep in Google Docs. Jamie recently added links to tho= se >> > > that he >> > > >> >>>>> could >> > > >> >>>>>>>> find >> > > >> >>>>>>>>>> on the mailing list to the Flink wiki: >> > > >> >>>>>>>>>> >> > > >> >>>> >> > > https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Home. >> > > >> >>>>>>>> The >> > > >> >>>>>>>>>> problem with these is that a) the comments on the Goog= le >> > Docs >> > > >> >>>> are not >> > > >> >>>>>>>>>> reflected in Jira and the mailing list. There has been >> some >> > > very >> > > >> >>>>>>> active >> > > >> >>>>>>>>>> discussion on some of the docs that most people would >> never >> > > >> >>>> notice. >> > > >> >>>>>>> The >> > > >> >>>>>>>>>> community therefore might seem less active than it >> actually >> > > is. >> > > >> >>>> b) >> > > >> >>>>> the >> > > >> >>>>>>>>>> documents are not very discoverable, if we had a clear= ly >> > > defined >> > > >> >>>>> place >> > > >> >>>>>>>>>> where we put them and also prominently link to this on >> the >> > > Flink >> > > >> >>>>>>>> homepage >> > > >> >>>>>>>>>> this would greatly help people that try to find out >> about >> > > >> current >> > > >> >>>>>>>>>> developments. >> > > >> >>>>>>>>>> >> > > >> >>>>>>>>>> Kafka has a process like this: >> > > >> >>>>>>>>>> >> > > >> >>>>>>>> >> > > >> >>>>>>> >> > > >> >>>>> >> > > >> >>>> >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Prop= osals >> > > >> >>>>>>>> . >> > > >> >>>>>>>>>> They call it KIP, for Kafka Improvement Proposal. We >> could >> > > >> either >> > > >> >>>>>>> adapt >> > > >> >>>>>>>>>> this for Flink or come up with our own process. Doing >> the >> > > former >> > > >> >>>>> would >> > > >> >>>>>>>> save >> > > >> >>>>>>>>>> us a lot of time and I don't think the Kafka community >> > would >> > > >> >>>> mind us >> > > >> >>>>>>>>>> copying their process. The subject also hints at this, >> our >> > > >> >>>> process >> > > >> >>>>>>>> could be >> > > >> >>>>>>>>>> called FLIP, for Flink Improvement Proposal. >> > > >> >>>>>>>>>> >> > > >> >>>>>>>>>> What do you think? Feedback is highly welcome. :-) >> > > >> >>>>>>>>>> >> > > >> >>>>>>>>>> Cheers, >> > > >> >>>>>>>>>> Aljoscha >> > > >> >>>>>>>>>> >> > > >> >>>>>>>>> >> > > >> >>>>>>>> >> > > >> >>>>>>> >> > > >> >>>>>> >> > > >> >>>>> >> > > >> >>>>> >> > > >> >>>> >> > > >> >>> >> > > >> > >> > > >> >> > > >> >> > > >> > >> > --001a1135b19453a785053775a6c1--