flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: [DISCUSS] Merge batch and stream connector modules
Date Thu, 22 Sep 2016 13:41:57 GMT
+1 for Fabian's suggestion

On Thu, Sep 22, 2016 at 3:25 PM, Swapnil Chougule <the.swapnil09@gmail.com>
wrote:

> +1
> It will be good to have one module flink-connectors (union of streaming and
> batch connectors).
>
> Regards,
> Swapnil
>
> On Thu, Sep 22, 2016 at 6:35 PM, Fabian Hueske <fhueske@gmail.com> wrote:
>
> > Hi everybody,
> >
> > right now, we have two separate Maven modules for batch and streaming
> > connectors (flink-batch-connectors and flink-streaming-connectors) that
> > contain modules for the individual external systems and storage formats
> > such as HBase, Cassandra, Avro, Elasticsearch, etc.
> >
> > Some of these systems can be used in streaming as well as batch jobs as
> for
> > instance HBase, Cassandra, and Elasticsearch. However, due to the
> separate
> > main modules for streaming and batch connectors, we currently need to
> > decide where to put a connector. For example, the
> flink-connector-cassandra
> > module is located in flink-streaming-connectors but includes a
> > CassandraInputFormat and CassandraOutputFormat (i.e., a batch source and
> > sink).
> >
> > In my opinion, it would be better to just merge flink-batch-connectors
> and
> > flink-streaming-connectors into a joint flink-connectors module.
> >
> > This would be only an internal restructuring of code and not be visible
> to
> > users (unless we change the module names of the individual connectors
> which
> > is not necessary, IMO).
> >
> > What do others think?
> >
> > Best, Fabian
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message