flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Saputra <henry.sapu...@gmail.com>
Subject Re: Externalizing the Flink connectors
Date Mon, 14 Dec 2015 19:16:45 GMT
Yes, that would be the way to go.

We could follow Cask CDAP hydrator plugin repository [1] that support
different plugins to run in their main CDAP hydrator [2]  product

- Henry

[1] https://github.com/caskdata/hydrator-plugins
[2] https://github.com/caskdata/cdap

On Mon, Dec 14, 2015 at 1:49 AM, Maximilian Michels <mxm@apache.org> wrote:
>>
>> Regarding Max suggestion to have version compatible connectors: I'm not
>> sure if we are able to maintain all connectors across different releases.
>>
>
> That was not my suggestion. Whenever we release, existing connectors should
> be compatible with that release. Otherwise, they should be removed from the
> release branch. This doesn't imply every connector version should be
> compatible across all releases.
>
> On Mon, Dec 14, 2015 at 10:39 AM, Robert Metzger <rmetzger@apache.org>
> wrote:
>> Regarding Max suggestion to have version compatible connectors: I'm not
>> sure if we are able to maintain all connectors across different releases.
> I
>> think its okay to have a document describing the minimum required Flink
>> version for each connector.
>>
>> With the interface stability guarantees from 1.0 on, the number of
> breaking
>> changes will go down.
>>
>> I'm against the name "plugins" because everything (documentation, code,
>> code comments, ...) is called "connectors" and it would be a pretty
>> breaking change. I also think that "connector" describes much better what
>> the whole thing is about.
>>
>>
>>
>> On Mon, Dec 14, 2015 at 10:20 AM, Maximilian Michels <mxm@apache.org>
> wrote:
>>
>>> Yes, absolutely. Setting up another repository for Flink ML would be no
>>> problem.
>>>
>>> On Sat, Dec 12, 2015 at 1:52 AM, Henry Saputra <henry.saputra@gmail.com>
>>> wrote:
>>> > I had small chat with Till about how to help manage Flink ML Libraries
>>> > contributions, which use Flink ML as dependencies.
>>> >
>>> > I suppose if this approached is the way to go for Flink connectors,
>>> > could we do the same for Flink ML libraries?
>>> >
>>> >
>>> > - Henry
>>> >
>>> > On Fri, Dec 11, 2015 at 1:33 AM, Maximilian Michels <mxm@apache.org>
>>> wrote:
>>> >> We should have release branches which are in sync with the release
>>> >> branches in the main repository. Connectors should be compatible
>>> >> across minor releases. The versioning could be of the form
>>> >> "flinkversion-connectorversion", e.g. 0.10-connector1.
>>> >>
>>> >>>The pluggable architecture is great! (why Don't we call it Flink
>>> plugins? my 2 cents)
>>> >>
>>> >> We can still change the name. IMHO "Plugins" is a bit broad since this
>>> >> is currently only targeted at the connectors included in Flink.
>>> >>
>>> >>>Would we loose test coverage by putting the connectors into a separate
>>> repository/maven project?
>>> >>
>>> >> Not necessarily. Two possibilities:
>>> >>
>>> >> 1) Run a connectors test jar during the normal Travis tests in the
>>> >> main repository
>>> >> 2) Trigger a Travis test run at the connectors repository upon a
>>> >> commit into the main repository
>>> >>
>>> >> Option 1 seems like the better alternative because we would
>>> >> immediately see if a change breaks the connectors. Of course, if
>>> >> changes are made in the connectors repository, we would also run tests
>>> >> with the main repository.
>>> >>
>>> >> On Thu, Dec 10, 2015 at 11:00 PM, jun aoki <jaoki@apache.org>
wrote:
>>> >>> The pluggable architecture is great! (why Don't we call it Flink
>>> plugins?
>>> >>> my 2 cents)
>>> >>> It will be nice if we come up with an idea of what directory
> structure
>>> >>> should look like before start dumping connectors (plugins).
>>> >>> Also wonder what to do with versioning.
>>> >>> At some point, for example, Twitter v1 connector could be compatible
>>> with
>>> >>> flink 0.10 but Flume v2 connector could be compatible with trunk,
> etc.
>>> It
>>> >>> should be taken consideration either in the directory structure
or
>>> >>> branching strategy.
>>> >>>
>>> >>> On Thu, Dec 10, 2015 at 7:12 AM, Aljoscha Krettek <
> aljoscha@apache.org
>>> >
>>> >>> wrote:
>>> >>>
>>> >>>> We would need to have a stable interface between the connectors
and
>>> flink
>>> >>>> and have very good checks that ensure that we don’t inadvertently
>>> break
>>> >>>> things.
>>> >>>>
>>> >>>> > On 10 Dec 2015, at 15:45, Fabian Hueske <fhueske@gmail.com>
wrote:
>>> >>>> >
>>> >>>> > Sounds like a good idea to me.
>>> >>>> >
>>> >>>> > +1
>>> >>>> >
>>> >>>> > Fabian
>>> >>>> >
>>> >>>> > 2015-12-10 15:31 GMT+01:00 Maximilian Michels <mxm@apache.org>:
>>> >>>> >
>>> >>>> >> Hi squirrels,
>>> >>>> >>
>>> >>>> >> By this time, we have numerous connectors which let
you insert
> data
>>> >>>> >> into Flink or output data from Flink.
>>> >>>> >>
>>> >>>> >> On the streaming side we have
>>> >>>> >>
>>> >>>> >> - RollingSink
>>> >>>> >> - Flume
>>> >>>> >> - Kafka
>>> >>>> >> - Nifi
>>> >>>> >> - RabbitMQ
>>> >>>> >> - Twitter
>>> >>>> >>
>>> >>>> >> On the batch side we have
>>> >>>> >>
>>> >>>> >> - Avro
>>> >>>> >> - Hadoop compatibility
>>> >>>> >> - HBase
>>> >>>> >> - HCatalog
>>> >>>> >> - JDBC
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> Many times we would have liked to release updates to
the
>>> connectors or
>>> >>>> >> even create new ones in between Flink releases. This
is currently
>>> not
>>> >>>> >> possible because the connectors are part of the main
repository.
>>> >>>> >>
>>> >>>> >> Therefore, I have created a new repository at
>>> >>>> >> https://git-wip-us.apache.org/repos/asf/flink-connectors.git.
The
>>> idea
>>> >>>> >> is to externalize the connectors to this repository.
We can then
>>> >>>> >> update and release them independently of the main Flink
>>> repository. I
>>> >>>> >> think this will give us more flexibility in the development
>>> process.
>>> >>>> >>
>>> >>>> >> What do you think about this idea?
>>> >>>> >>
>>> >>>> >> Cheers,
>>> >>>> >> Max
>>> >>>> >>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> -jun
>>>

Mime
View raw message