Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <CANC1h_savjX7q7SBa=kWqaJd9iVJM6WTz2wVq7m+x6Q7QGN12w@mail.gmail.com>
References: <CAGr9p8CAN8KQTM6+3+=Nv8M3ggYEE9gSqdKaKLQiWsWsKzZ21Q@mail.gmail.com>
 <CAJnSTVyTuaLGPoYxXrmp_kj5Ha=QQn+gmmz6gxtRDge+dK826Q@mail.gmail.com>
 <CAGr9p8B_mgCeokE=cOijXYaSkok6_iPFC7-J8A8yqaB4W++vfA@mail.gmail.com>
 <CAJnSTVxTitsXHsRm8n5xwEVvvAvggZK_0mJ1qo_8EnP-E4GO7Q@mail.gmail.com>
 <CANC1h_s9543CuRd5n8nnMmsTB7_s_Xyn=4B-N2-ALbS+Vn-udw@mail.gmail.com>
 <CAC27z=MQf8rVK3A5GdDNK3G595HM4BwnZpPTMXV0zYKWDaVing@mail.gmail.com>
 <CAGr9p8AcX5EQ5FVHP6kVshBxbV9yx9jBV1fwv4puMu_Wau2ykg@mail.gmail.com>
 <CAGco--Zshbe++K2zAx=5VgeR4=wyNVy3W5jPgnDd_tpnrRM=yA@mail.gmail.com>
 <CAGr9p8AOPt_qkqJRRnrdLBQLHq8e0n0C1CLA4iRXXyd4T2=kdA@mail.gmail.com>
 <CAGr9p8CeVrOyAnvwV6cX-_9K5XvuqZbBW7McuvNBS-Pz8WDGBw@mail.gmail.com> <CANC1h_savjX7q7SBa=kWqaJd9iVJM6WTz2wVq7m+x6Q7QGN12w@mail.gmail.com>
From: Robert Metzger <rmetzger@apache.org>
Date: Wed, 17 Aug 2016 17:24:34 +0200
Message-ID: <CAGr9p8A8A8pX+zkbcXfvUbhKuODtDkVR6-TpnX0k4y-K6MkK9g@mail.gmail.com>
Subject: Re: [DISCUSS] Streaming connector contributions
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a1130c92e448acf053a4611d0
archived-at: Wed, 17 Aug 2016 15:24:59 -0000

--001a1130c92e448acf053a4611d0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Okay, I agree that this is something we should make very clear.

I'll start a DISCUSS thread.

On Wed, Aug 17, 2016 at 10:07 AM, Stephan Ewen <sewen@apache.org> wrote:

> Cool to see that the Bahir people are so open and helpful!
>
> Concerning moving the connectors: I think we should set up a vote thread,
> or at least a discussion with the possibility to object.
> Removing someones code from the repository is a bit of a sensitive thing =
in
> my experience, so let's make sure everybody is behind that decision befor=
e
> we proceed.
>
> On Wed, Aug 17, 2016 at 9:31 AM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
> > Bahir is now creating a second repository "bahir-flink" for our
> connectors:
> > https://issues.apache.org/jira/browse/INFRA-12440
> >
> > If there are no objections here on the dev list, I would like to move t=
he
> > "flume" and "redis" streaming connector to Bahir.
> > Once they are in, and the file structure at Bahir exists, I'll update o=
ur
> > contribution guidelines to point people to Bahir.
> >
> > On Mon, Aug 15, 2016 at 2:14 PM, Robert Metzger <rmetzger@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I just wanted to let you know that I've started a discussion at the
> Bahir
> > > dev list [1]. They seem to be very open to give some of our streaming
> > > connectors a new home.
> > > We are currently discussing some specifics there (whether to put the
> code
> > > into the same repository or into separate ones). Also, they asked for
> > some
> > > help by Flink committers / contributors to help with the reviewing of
> > > incoming contributions.
> > > I'm definitively willing to help out to give the community there a go=
od
> > > start.
> > >
> > >
> > > [1] https://www.mail-archive.com/dev@bahir.apache.org/msg00357.html
> > >
> > >
> > > On Fri, Aug 12, 2016 at 10:54 AM, Maximilian Michels <mxm@apache.org>
> > > wrote:
> > >
> > >> Hi Robert,
> > >>
> > >> We had this discussion before when I suggested to use an external
> > >> repository to manage connectors. Ever since I have come to the
> > >> conclusion that the overhead of maintaining two source repositories
> > >> along with maintaining code and integration, documentation, and CI, =
is
> > >> not worth the effort. Thanks for bringing up the discussion again.
> > >>
> > >> I think the most important issues with moving connectors out of the
> > >> core repository are
> > >>
> > >> 1) Maintenance
> > >> Connectors need to be maintained. Otherwise, they are worthless.
> > >>
> > >> 2) Plugability
> > >> Connectors need to be easy to use for the user. Otherwise, nobody wi=
ll
> > >> use them.
> > >>
> > >> What is the difference between #1, #3 and #4? I don't see a differen=
ce
> > >> except that #3 and #4 are central repositories. Still, maintenance a=
nd
> > >> plugability are very unclear.
> > >>
> > >> Only #2 is left :) Apache Bahir looks like a promising project. Let'=
s
> > >> see how the community reacts.
> > >>
> > >> Cheers,
> > >> Max
> > >>
> > >>
> > >> On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger <rmetzger@apache.or=
g
> >
> > >> wrote:
> > >> > Thank you for your responses. I will get in touch with the Bahir
> > >> community
> > >> > to see what they are thinking about this.
> > >> > Once we know a bit more about the details of such a collaboration,
> we
> > >> can
> > >> > make a final decision here.
> > >> >
> > >> > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann <trohrmann@apache.or=
g
> >
> > >> wrote:
> > >> >
> > >> >> I agree with Stephan that the main problem is maintenance overhea=
d
> > for
> > >> the
> > >> >> Flink community. If we could maintain all connectors ourselves th=
en
> > >> there
> > >> >> would not be an immediate need to out source the connectors. Thus=
,
> > the
> > >> >> solution should reduce the workload for the core project.
> > >> >>
> > >> >> Personally, I would favour option #2 if Apache Bahir is willing t=
o
> > add
> > >> >> Flink as another supported streaming platform. This would give th=
e
> > >> >> streaming connectors a more prominent home than the personal
> > >> repository of
> > >> >> a contributor.
> > >> >>
> > >> >> Furthermore, contributing the connectors to Apache Bahir entails
> that
> > >> the
> > >> >> connector artifacts are deployed to a central repository (I
> assume).
> > >> That
> > >> >> way one can more easily add them to one's project instead of havi=
ng
> > to
> > >> >> build and install the code yourself.
> > >> >>
> > >> >> I'm also not sure what happens to a public github repository whic=
h
> > has
> > >> not
> > >> >> been forked and is then deleted. I would assume that the content =
is
> > >> then
> > >> >> lost.
> > >> >>
> > >> >> To create a "flink-connectors" repository independent of Flink
> sounds
> > >> quite
> > >> >> similar to creating another Apache Bahir. I think it would be
> better
> > to
> > >> >> join forces with the existing Bahir community if possible.
> > >> >>
> > >> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen <sewen@apache.org>
> > wrote:
> > >> >>
> > >> >> > Thanks Robert for bringing this up.
> > >> >> >
> > >> >> > I think that "flink-contrib" will not really solve the problem,
> > >> because
> > >> >> the
> > >> >> > connectors would still have to be maintained by the core
> community,
> > >> we
> > >> >> > would need to guarantee test stability. It will be to a large
> > extend
> > >> >> > similar to adding them to "flink-streaming-connectors", except
> that
> > >> we
> > >> >> may
> > >> >> > declare via the artifact name that the quality is not guarantee=
d
> to
> > >> be
> > >> >> too
> > >> >> > high.
> > >> >> >
> > >> >> > Regarding moving the code to a separate repository: I think tha=
t
> > also
> > >> >> only
> > >> >> > solves the problem, if that repository is organizationally
> > different
> > >> from
> > >> >> > the core clink repository. Different release cycles, different
> > >> >> maintainers.
> > >> >> >
> > >> >> >
> > >> >> > The quintessence is for me that the connectors need to be handl=
ed
> > by
> > >> a
> > >> >> > different organization than the core flink community.
> > >> >> > That would leave us with
> > >> >> >   - The contributors' own repository
> > >> >> >   - Apache Bahir
> > >> >> >   - An organizationally different "flink-connectors" repository=
,
> > >> possibly
> > >> >> > with different committers / PMC
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai <
> > >> tzulitai@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Good point. Discussion for each new connector is also an
> overhead
> > >> we
> > >> >> > should
> > >> >> > > avoid.
> > >> >> > >
> > >> >> > > IMHO, option #2 doesn=E2=80=99t seem like a reasonable stagin=
g area.
> Say
> > we
> > >> >> > decide
> > >> >> > > we=E2=80=99d like to move a connector from Bahir to Flink in =
the
> future,
> > >> >> there=E2=80=99ll
> > >> >> > > be two of the connector in separate Apache projects.
> Personally I
> > >> think
> > >> >> > > option #2 is more like a way to go if some day we=E2=80=99d l=
ike to
> > migrate
> > >> >> some
> > >> >> > of
> > >> >> > > our currently supported connectors in Flink to Bahir (perhaps
> an
> > >> >> > > alternative to the discussion in "Externalizing the Flink
> > >> connectors=E2=80=9D).
> > >> >> > >
> > >> >> > > Defining option #1 as a staging area seems nice; the
> contributor
> > >> >> notifies
> > >> >> > > the Flink community of the work by adding it to the 3rd party
> > >> packages
> > >> >> > > page, and in the future we can contact the contributor to ask
> > >> whether
> > >> >> > > they=E2=80=99d like to migrate the connector to Flink when we=
 see more
> > user
> > >> >> > demand
> > >> >> > > and committer support.
> > >> >> > >
> > >> >> > > With this approach, do you think we should assume that future
> new
> > >> >> > > connectors always go to option #1 first?   If not, I would
> assume
> > >> that
> > >> >> in
> > >> >> > > the future, Flink JIRAs for new connectors are only opened if
> > we=E2=80=99d
> > >> like
> > >> >> > to
> > >> >> > > add them directly to Flink (perhaps the contribution guidelin=
e
> > can
> > >> link
> > >> >> > to
> > >> >> > > Flink development Roadmaps like [1] as a reference to
> connectors
> > >> that
> > >> >> the
> > >> >> > > community has already considered to support?).
> > >> >> > >
> > >> >> > > [1]
> > >> >> > > https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JK
> Xm-
> > >> >> > > 778DAD7eqw5GANwE/edit#
> > >> >> > > <https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5J
> KXm-
> > >> >> > > 778DAD7eqw5GANwE/edit>
> > >> >> > >
> > >> >> > > On August 9, 2016 at 6:10:21 PM, Robert Metzger (
> > >> rmetzger@apache.org)
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > Hi Gordon,
> > >> >> > > thank you for your response.
> > >> >> > >
> > >> >> > > I agree with your observation that some "staging" area is
> helpful
> > >> to
> > >> >> test
> > >> >> > > how many contributors / users are interested in a connector.
> But
> > I
> > >> >> wonder
> > >> >> > > if #1 or #2 can also serve as a staging area: As soon as we s=
ee
> > >> that
> > >> >> > there
> > >> >> > > is a lot of interest in a connector, we add it to the main
> > >> >> > > "flink-*-connectors" directory.
> > >> >> > > Having too many ways to contribute connectors might make the
> > >> >> discussions
> > >> >> > > for each new connector quite complicated.
> > >> >> > >
> > >> >> > > You are right, the intention of the "Externalizing the Flink
> > >> >> connectors=E2=80=9D
> > >> >> > > discussion was more about the release intervals, test time et=
c.
> > >> >> > >
> > >> >> > > Regards,
> > >> >> > > Robert
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> > >
> > >
> >
>

--001a1130c92e448acf053a4611d0--