flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: [DISCUSS] Streaming connector contributions
Date Wed, 17 Aug 2016 08:07:36 GMT
Cool to see that the Bahir people are so open and helpful!

Concerning moving the connectors: I think we should set up a vote thread,
or at least a discussion with the possibility to object.
Removing someones code from the repository is a bit of a sensitive thing in
my experience, so let's make sure everybody is behind that decision before
we proceed.

On Wed, Aug 17, 2016 at 9:31 AM, Robert Metzger <rmetzger@apache.org> wrote:

> Bahir is now creating a second repository "bahir-flink" for our connectors:
> https://issues.apache.org/jira/browse/INFRA-12440
>
> If there are no objections here on the dev list, I would like to move the
> "flume" and "redis" streaming connector to Bahir.
> Once they are in, and the file structure at Bahir exists, I'll update our
> contribution guidelines to point people to Bahir.
>
> On Mon, Aug 15, 2016 at 2:14 PM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
> > Hi,
> >
> > I just wanted to let you know that I've started a discussion at the Bahir
> > dev list [1]. They seem to be very open to give some of our streaming
> > connectors a new home.
> > We are currently discussing some specifics there (whether to put the code
> > into the same repository or into separate ones). Also, they asked for
> some
> > help by Flink committers / contributors to help with the reviewing of
> > incoming contributions.
> > I'm definitively willing to help out to give the community there a good
> > start.
> >
> >
> > [1] https://www.mail-archive.com/dev@bahir.apache.org/msg00357.html
> >
> >
> > On Fri, Aug 12, 2016 at 10:54 AM, Maximilian Michels <mxm@apache.org>
> > wrote:
> >
> >> Hi Robert,
> >>
> >> We had this discussion before when I suggested to use an external
> >> repository to manage connectors. Ever since I have come to the
> >> conclusion that the overhead of maintaining two source repositories
> >> along with maintaining code and integration, documentation, and CI, is
> >> not worth the effort. Thanks for bringing up the discussion again.
> >>
> >> I think the most important issues with moving connectors out of the
> >> core repository are
> >>
> >> 1) Maintenance
> >> Connectors need to be maintained. Otherwise, they are worthless.
> >>
> >> 2) Plugability
> >> Connectors need to be easy to use for the user. Otherwise, nobody will
> >> use them.
> >>
> >> What is the difference between #1, #3 and #4? I don't see a difference
> >> except that #3 and #4 are central repositories. Still, maintenance and
> >> plugability are very unclear.
> >>
> >> Only #2 is left :) Apache Bahir looks like a promising project. Let's
> >> see how the community reacts.
> >>
> >> Cheers,
> >> Max
> >>
> >>
> >> On Thu, Aug 11, 2016 at 10:26 AM, Robert Metzger <rmetzger@apache.org>
> >> wrote:
> >> > Thank you for your responses. I will get in touch with the Bahir
> >> community
> >> > to see what they are thinking about this.
> >> > Once we know a bit more about the details of such a collaboration, we
> >> can
> >> > make a final decision here.
> >> >
> >> > On Tue, Aug 9, 2016 at 3:47 PM, Till Rohrmann <trohrmann@apache.org>
> >> wrote:
> >> >
> >> >> I agree with Stephan that the main problem is maintenance overhead
> for
> >> the
> >> >> Flink community. If we could maintain all connectors ourselves then
> >> there
> >> >> would not be an immediate need to out source the connectors. Thus,
> the
> >> >> solution should reduce the workload for the core project.
> >> >>
> >> >> Personally, I would favour option #2 if Apache Bahir is willing to
> add
> >> >> Flink as another supported streaming platform. This would give the
> >> >> streaming connectors a more prominent home than the personal
> >> repository of
> >> >> a contributor.
> >> >>
> >> >> Furthermore, contributing the connectors to Apache Bahir entails that
> >> the
> >> >> connector artifacts are deployed to a central repository (I assume).
> >> That
> >> >> way one can more easily add them to one's project instead of having
> to
> >> >> build and install the code yourself.
> >> >>
> >> >> I'm also not sure what happens to a public github repository which
> has
> >> not
> >> >> been forked and is then deleted. I would assume that the content is
> >> then
> >> >> lost.
> >> >>
> >> >> To create a "flink-connectors" repository independent of Flink sounds
> >> quite
> >> >> similar to creating another Apache Bahir. I think it would be better
> to
> >> >> join forces with the existing Bahir community if possible.
> >> >>
> >> >> On Tue, Aug 9, 2016 at 2:56 PM, Stephan Ewen <sewen@apache.org>
> wrote:
> >> >>
> >> >> > Thanks Robert for bringing this up.
> >> >> >
> >> >> > I think that "flink-contrib" will not really solve the problem,
> >> because
> >> >> the
> >> >> > connectors would still have to be maintained by the core community,
> >> we
> >> >> > would need to guarantee test stability. It will be to a large
> extend
> >> >> > similar to adding them to "flink-streaming-connectors", except
that
> >> we
> >> >> may
> >> >> > declare via the artifact name that the quality is not guaranteed
to
> >> be
> >> >> too
> >> >> > high.
> >> >> >
> >> >> > Regarding moving the code to a separate repository: I think that
> also
> >> >> only
> >> >> > solves the problem, if that repository is organizationally
> different
> >> from
> >> >> > the core clink repository. Different release cycles, different
> >> >> maintainers.
> >> >> >
> >> >> >
> >> >> > The quintessence is for me that the connectors need to be handled
> by
> >> a
> >> >> > different organization than the core flink community.
> >> >> > That would leave us with
> >> >> >   - The contributors' own repository
> >> >> >   - Apache Bahir
> >> >> >   - An organizationally different "flink-connectors" repository,
> >> possibly
> >> >> > with different committers / PMC
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Tue, Aug 9, 2016 at 1:41 PM, Tzu-Li (Gordon) Tai <
> >> tzulitai@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > > Good point. Discussion for each new connector is also an
overhead
> >> we
> >> >> > should
> >> >> > > avoid.
> >> >> > >
> >> >> > > IMHO, option #2 doesn’t seem like a reasonable staging
area. Say
> we
> >> >> > decide
> >> >> > > we’d like to move a connector from Bahir to Flink in the
future,
> >> >> there’ll
> >> >> > > be two of the connector in separate Apache projects. Personally
I
> >> think
> >> >> > > option #2 is more like a way to go if some day we’d like
to
> migrate
> >> >> some
> >> >> > of
> >> >> > > our currently supported connectors in Flink to Bahir (perhaps
an
> >> >> > > alternative to the discussion in "Externalizing the Flink
> >> connectors”).
> >> >> > >
> >> >> > > Defining option #1 as a staging area seems nice; the contributor
> >> >> notifies
> >> >> > > the Flink community of the work by adding it to the 3rd party
> >> packages
> >> >> > > page, and in the future we can contact the contributor to
ask
> >> whether
> >> >> > > they’d like to migrate the connector to Flink when we see
more
> user
> >> >> > demand
> >> >> > > and committer support.
> >> >> > >
> >> >> > > With this approach, do you think we should assume that future
new
> >> >> > > connectors always go to option #1 first?   If not, I would
assume
> >> that
> >> >> in
> >> >> > > the future, Flink JIRAs for new connectors are only opened
if
> we’d
> >> like
> >> >> > to
> >> >> > > add them directly to Flink (perhaps the contribution guideline
> can
> >> link
> >> >> > to
> >> >> > > Flink development Roadmaps like [1] as a reference to connectors
> >> that
> >> >> the
> >> >> > > community has already considered to support?).
> >> >> > >
> >> >> > > [1]
> >> >> > > https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-
> >> >> > > 778DAD7eqw5GANwE/edit#
> >> >> > > <https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm-
> >> >> > > 778DAD7eqw5GANwE/edit>
> >> >> > >
> >> >> > > On August 9, 2016 at 6:10:21 PM, Robert Metzger (
> >> rmetzger@apache.org)
> >> >> > > wrote:
> >> >> > >
> >> >> > > Hi Gordon,
> >> >> > > thank you for your response.
> >> >> > >
> >> >> > > I agree with your observation that some "staging" area is
helpful
> >> to
> >> >> test
> >> >> > > how many contributors / users are interested in a connector.
But
> I
> >> >> wonder
> >> >> > > if #1 or #2 can also serve as a staging area: As soon as
we see
> >> that
> >> >> > there
> >> >> > > is a lot of interest in a connector, we add it to the main
> >> >> > > "flink-*-connectors" directory.
> >> >> > > Having too many ways to contribute connectors might make
the
> >> >> discussions
> >> >> > > for each new connector quite complicated.
> >> >> > >
> >> >> > > You are right, the intention of the "Externalizing the Flink
> >> >> connectors”
> >> >> > > discussion was more about the release intervals, test time
etc.
> >> >> > >
> >> >> > > Regards,
> >> >> > > Robert
> >> >> > >
> >> >> >
> >> >>
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message