flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: [DISCUSS] Merge batch and stream connector modules
Date Fri, 23 Sep 2016 09:15:33 GMT
I don't think it's that easy. The streaming connectors have flink-streaming
as dependency while the batch connectors have the batch dependencies.

Combining them would mean that users always have all dependencies, right?

On Thu, 22 Sep 2016 at 15:41 Stephan Ewen <sewen@apache.org> wrote:

> +1 for Fabian's suggestion
>
> On Thu, Sep 22, 2016 at 3:25 PM, Swapnil Chougule <the.swapnil09@gmail.com
> >
> wrote:
>
> > +1
> > It will be good to have one module flink-connectors (union of streaming
> and
> > batch connectors).
> >
> > Regards,
> > Swapnil
> >
> > On Thu, Sep 22, 2016 at 6:35 PM, Fabian Hueske <fhueske@gmail.com>
> wrote:
> >
> > > Hi everybody,
> > >
> > > right now, we have two separate Maven modules for batch and streaming
> > > connectors (flink-batch-connectors and flink-streaming-connectors) that
> > > contain modules for the individual external systems and storage formats
> > > such as HBase, Cassandra, Avro, Elasticsearch, etc.
> > >
> > > Some of these systems can be used in streaming as well as batch jobs as
> > for
> > > instance HBase, Cassandra, and Elasticsearch. However, due to the
> > separate
> > > main modules for streaming and batch connectors, we currently need to
> > > decide where to put a connector. For example, the
> > flink-connector-cassandra
> > > module is located in flink-streaming-connectors but includes a
> > > CassandraInputFormat and CassandraOutputFormat (i.e., a batch source
> and
> > > sink).
> > >
> > > In my opinion, it would be better to just merge flink-batch-connectors
> > and
> > > flink-streaming-connectors into a joint flink-connectors module.
> > >
> > > This would be only an internal restructuring of code and not be visible
> > to
> > > users (unless we change the module names of the individual connectors
> > which
> > > is not necessary, IMO).
> > >
> > > What do others think?
> > >
> > > Best, Fabian
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message