flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: [DISCUSS] Streaming connector contributions
Date Wed, 17 Aug 2016 15:24:34 GMT
Okay, I agree that this is something we should make very clear.

I'll start a DISCUSS thread.

On Wed, Aug 17, 2016 at 10:07 AM, Stephan Ewen <sewen@apache.org> wrote:

> Cool to see that the Bahir people are so open and helpful!
>
> Concerning moving the connectors: I think we should set up a vote thread,
> or at least a discussion with the possibility to object.
> Removing someones code from the repository is a bit of a sensitive thing in
> my experience, so let's make sure everybody is behind that decision before
> we proceed.
>
> On Wed, Aug 17, 2016 at 9:31 AM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
> > Bahir is now creating a second repository "bahir-flink" for our
> connectors:
> > https://issues.apache.org/jira/browse/INFRA-12440
> >
> > If there are no objections here on the dev list, I would like to move the
> > "flume" and "redis" streaming connector to Bahir.
> > Once they are in, and the file structure at Bahir exists, I'll update our
> > contribution guidelines to point people to Bahir.
> >
> > On Mon, Aug 15, 2016 at 2:14 PM, Robert Metzger <rmetzger@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I just wanted to let you know that I've started a discussion at the
> Bahir
> > > dev list [1]. They seem to be very open to give some of our streaming
> > > connectors a new home.
> > > We are currently discussing some specifics there (whether to put the
> code
> > > into the same repository or into separate ones). Also, they asked for
> > some
> > > help by Flink committers / contributors to help with the reviewing of
> > > incoming contributions.
> > > I'm definitively willing to help out to give the community there a good
> > > start.
> > >
> > >
> > > [1] https://www.mail-archive.com/dev@bahir.apache.org/msg00357.html
> > >
> > >
> > > On Fri, Aug 12, 2016 at 10:54 AM, Maximilian Michels <mxm@apache.org>
> > > wrote:
> > >
> > >> Hi Robert,
> > >>
> > >> We had this discussion before when I suggested to use an external
> > >> repository to manage connectors. Ever since I have come to the
> > >> conclusion that the overhead of maintaining two source repositories
> > >> along with maintaining code and integration, documentation, and CI, is
> > >> not worth the effort. Thanks for bringing up the discussion again.
> > >>
> > >> I think the most important issues with moving connectors out of the
> > >> core repository are
> > >>
> > >> 1) Maintenance
> > >> Connectors need to be maintained. Otherwise, they are worthless.
> > >>
> > >> 2) Plugability
> > >> Connectors need to be easy to use for the user. Otherwise, nobody will
> > >> use them.
> > >>
> > >> What is the difference between #1, #3 and #4? I don't see a difference
> > >> except that #3 and #4 are central repositories. Still, maintenance and
> > >> plugability are very unclear.
> > >>
> > >> Only #2 is left :) Apache Bahir looks like a promising project. Let's
> > >> see how the community reacts.
> > >>
> > >> Cheers,
> > >> Max
> > >>
> > >>
> > >> On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger <rmetzger@apache.org
> >
> > >> wrote:
> > >> > Thank you for your responses. I will get in touch with the Bahir
> > >> community
> > >> > to see what they are thinking about this.
> > >> > Once we know a bit more about the details of such a collaboration,
> we
> > >> can
> > >> > make a final decision here.
> > >> >
> > >> > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann <trohrmann@apache.org
> >
> > >> wrote:
> > >> >
> > >> >> I agree with Stephan that the main problem is maintenance overhead
> > for
> > >> the
> > >> >> Flink community. If we could maintain all connectors ourselves
then
> > >> there
> > >> >> would not be an immediate need to out source the connectors. Thus,
> > the
> > >> >> solution should reduce the workload for the core project.
> > >> >>
> > >> >> Personally, I would favour option #2 if Apache Bahir is willing
to
> > add
> > >> >> Flink as another supported streaming platform. This would give
the
> > >> >> streaming connectors a more prominent home than the personal
> > >> repository of
> > >> >> a contributor.
> > >> >>
> > >> >> Furthermore, contributing the connectors to Apache Bahir entails
> that
> > >> the
> > >> >> connector artifacts are deployed to a central repository (I
> assume).
> > >> That
> > >> >> way one can more easily add them to one's project instead of having
> > to
> > >> >> build and install the code yourself.
> > >> >>
> > >> >> I'm also not sure what happens to a public github repository which
> > has
> > >> not
> > >> >> been forked and is then deleted. I would assume that the content
is
> > >> then
> > >> >> lost.
> > >> >>
> > >> >> To create a "flink-connectors" repository independent of Flink
> sounds
> > >> quite
> > >> >> similar to creating another Apache Bahir. I think it would be
> better
> > to
> > >> >> join forces with the existing Bahir community if possible.
> > >> >>
> > >> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen <sewen@apache.org>
> > wrote:
> > >> >>
> > >> >> > Thanks Robert for bringing this up.
> > >> >> >
> > >> >> > I think that "flink-contrib" will not really solve the problem,
> > >> because
> > >> >> the
> > >> >> > connectors would still have to be maintained by the core
> community,
> > >> we
> > >> >> > would need to guarantee test stability. It will be to a large
> > extend
> > >> >> > similar to adding them to "flink-streaming-connectors", except
> that
> > >> we
> > >> >> may
> > >> >> > declare via the artifact name that the quality is not guaranteed
> to
> > >> be
> > >> >> too
> > >> >> > high.
> > >> >> >
> > >> >> > Regarding moving the code to a separate repository: I think
that
> > also
> > >> >> only
> > >> >> > solves the problem, if that repository is organizationally
> > different
> > >> from
> > >> >> > the core clink repository. Different release cycles, different
> > >> >> maintainers.
> > >> >> >
> > >> >> >
> > >> >> > The quintessence is for me that the connectors need to be
handled
> > by
> > >> a
> > >> >> > different organization than the core flink community.
> > >> >> > That would leave us with
> > >> >> >   - The contributors' own repository
> > >> >> >   - Apache Bahir
> > >> >> >   - An organizationally different "flink-connectors" repository,
> > >> possibly
> > >> >> > with different committers / PMC
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai <
> > >> tzulitai@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > Good point. Discussion for each new connector is also
an
> overhead
> > >> we
> > >> >> > should
> > >> >> > > avoid.
> > >> >> > >
> > >> >> > > IMHO, option #2 doesn’t seem like a reasonable staging
area.
> Say
> > we
> > >> >> > decide
> > >> >> > > we’d like to move a connector from Bahir to Flink
in the
> future,
> > >> >> there’ll
> > >> >> > > be two of the connector in separate Apache projects.
> Personally I
> > >> think
> > >> >> > > option #2 is more like a way to go if some day we’d
like to
> > migrate
> > >> >> some
> > >> >> > of
> > >> >> > > our currently supported connectors in Flink to Bahir
(perhaps
> an
> > >> >> > > alternative to the discussion in "Externalizing the
Flink
> > >> connectors”).
> > >> >> > >
> > >> >> > > Defining option #1 as a staging area seems nice; the
> contributor
> > >> >> notifies
> > >> >> > > the Flink community of the work by adding it to the
3rd party
> > >> packages
> > >> >> > > page, and in the future we can contact the contributor
to ask
> > >> whether
> > >> >> > > they’d like to migrate the connector to Flink when
we see more
> > user
> > >> >> > demand
> > >> >> > > and committer support.
> > >> >> > >
> > >> >> > > With this approach, do you think we should assume that
future
> new
> > >> >> > > connectors always go to option #1 first?   If not, I
would
> assume
> > >> that
> > >> >> in
> > >> >> > > the future, Flink JIRAs for new connectors are only
opened if
> > we’d
> > >> like
> > >> >> > to
> > >> >> > > add them directly to Flink (perhaps the contribution
guideline
> > can
> > >> link
> > >> >> > to
> > >> >> > > Flink development Roadmaps like [1] as a reference to
> connectors
> > >> that
> > >> >> the
> > >> >> > > community has already considered to support?).
> > >> >> > >
> > >> >> > > [1]
> > >> >> > > https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JK
> Xm-
> > >> >> > > 778DAD7eqw5GANwE/edit#
> > >> >> > > <https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5J
> KXm-
> > >> >> > > 778DAD7eqw5GANwE/edit>
> > >> >> > >
> > >> >> > > On August 9, 2016 at 6:10:21 PM, Robert Metzger (
> > >> rmetzger@apache.org)
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > Hi Gordon,
> > >> >> > > thank you for your response.
> > >> >> > >
> > >> >> > > I agree with your observation that some "staging" area
is
> helpful
> > >> to
> > >> >> test
> > >> >> > > how many contributors / users are interested in a connector.
> But
> > I
> > >> >> wonder
> > >> >> > > if #1 or #2 can also serve as a staging area: As soon
as we see
> > >> that
> > >> >> > there
> > >> >> > > is a lot of interest in a connector, we add it to the
main
> > >> >> > > "flink-*-connectors" directory.
> > >> >> > > Having too many ways to contribute connectors might
make the
> > >> >> discussions
> > >> >> > > for each new connector quite complicated.
> > >> >> > >
> > >> >> > > You are right, the intention of the "Externalizing the
Flink
> > >> >> connectors”
> > >> >> > > discussion was more about the release intervals, test
time etc.
> > >> >> > >
> > >> >> > > Regards,
> > >> >> > > Robert
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message