flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Márton Balassi <balassi.mar...@gmail.com>
Subject Re: [DISCUSS] Break up streaming connectors into subprojects
Date Wed, 08 Apr 2015 13:00:11 GMT
Overall I think this is a nice approach, but let us then also discuss where
would we like to put these jars. Currently these jars are not in the lib
folder of the Flink distribution, which mean that whenever a user would
like to use them they have to package it with there usercode which is a bit
intuitive I think as they are in the org.apache.flink namespace.

The current approach was perfectly fine a month ago, when the connectors
where practically examples and not really connectors in the sense that we
actually expect users to use these exact classes as entry points to message
qeues. Now with the new PersistentKafkaSource I'm not quite sure that this
should be the case. Of course one can argue that these modules inherently
pull a lot of dependencies (Kafka, Zookeeper etc.) so it is better to avoid
them.

If we decide to add the connectors to the lib of the distribution then it
is not much use to separate them. If not then I support to do it.

On Wed, Apr 8, 2015 at 1:33 PM, Stephan Ewen <sewen@apache.org> wrote:

> Exactly each streaming connector would be a separate jar:
>
>   - stream-connector-kafka
>   - stream-connector-rabbitmq
>   - stream-connector-flume
>   - ...
>
> On Tue, Apr 7, 2015 at 10:59 PM, Henry Saputra <henry.saputra@gmail.com>
> wrote:
>
> > Would this proposal also include packaging streaming connectors into
> > separate source and binary jars?
> >
> > - Henry
> >
> > On Tue, Apr 7, 2015 at 12:21 PM, Stephan Ewen <sewen@apache.org> wrote:
> > > What do you think about dividing the streaming connectors project into
> > > various smaller projects, basically one per connector?
> > >
> > > I am personally always happy when projects offer me artifacts that
> > contain
> > > what I need, and not a lot of other unnecessary dependencies as well
> > >
> > > Many people using the streaming connectors as a dependency in other
> > setups
> > > will have to define a long list of exclusions, to get rid of all
> > > dependencies (and their transitives) that they do not need.
> > >
> > > We have seen how these "super fat" dependencies cause trouble, for
> > example
> > > at the case of Hadoop 1 where everything was one artifact and how much
> > > easier it is with Hadoop 2 where we can use subcomponents as
> > dependencies.
> > >
> > > What do you think?
> > >
> > > Stephan
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message