flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: [DISCUSS] Break up streaming connectors into subprojects
Date Thu, 09 Apr 2015 14:08:14 GMT
Thanks for bringing up this discussion.

I'm very much in favor of splitting up the connectors into separate maven
modules.
The transitive dependencies are a mess otherwise.

Also, I would not put them to "flink-dist" (=lib folder) because we would
have the dependency mess again.


On Wed, Apr 8, 2015 at 3:00 PM, Márton Balassi <balassi.marton@gmail.com>
wrote:

> Overall I think this is a nice approach, but let us then also discuss where
> would we like to put these jars. Currently these jars are not in the lib
> folder of the Flink distribution, which mean that whenever a user would
> like to use them they have to package it with there usercode which is a bit
> intuitive I think as they are in the org.apache.flink namespace.
>
> The current approach was perfectly fine a month ago, when the connectors
> where practically examples and not really connectors in the sense that we
> actually expect users to use these exact classes as entry points to message
> qeues. Now with the new PersistentKafkaSource I'm not quite sure that this
> should be the case. Of course one can argue that these modules inherently
> pull a lot of dependencies (Kafka, Zookeeper etc.) so it is better to avoid
> them.
>
> If we decide to add the connectors to the lib of the distribution then it
> is not much use to separate them. If not then I support to do it.
>
> On Wed, Apr 8, 2015 at 1:33 PM, Stephan Ewen <sewen@apache.org> wrote:
>
> > Exactly each streaming connector would be a separate jar:
> >
> >   - stream-connector-kafka
> >   - stream-connector-rabbitmq
> >   - stream-connector-flume
> >   - ...
> >
> > On Tue, Apr 7, 2015 at 10:59 PM, Henry Saputra <henry.saputra@gmail.com>
> > wrote:
> >
> > > Would this proposal also include packaging streaming connectors into
> > > separate source and binary jars?
> > >
> > > - Henry
> > >
> > > On Tue, Apr 7, 2015 at 12:21 PM, Stephan Ewen <sewen@apache.org>
> wrote:
> > > > What do you think about dividing the streaming connectors project
> into
> > > > various smaller projects, basically one per connector?
> > > >
> > > > I am personally always happy when projects offer me artifacts that
> > > contain
> > > > what I need, and not a lot of other unnecessary dependencies as well
> > > >
> > > > Many people using the streaming connectors as a dependency in other
> > > setups
> > > > will have to define a long list of exclusions, to get rid of all
> > > > dependencies (and their transitives) that they do not need.
> > > >
> > > > We have seen how these "super fat" dependencies cause trouble, for
> > > example
> > > > at the case of Hadoop 1 where everything was one artifact and how
> much
> > > > easier it is with Hadoop 2 where we can use subcomponents as
> > > dependencies.
> > > >
> > > > What do you think?
> > > >
> > > > Stephan
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message