flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: [DISCUSS] flink-external
Date Thu, 08 Oct 2015 10:28:18 GMT
Thanks for the feedback.

I think, the repository does not need to build on a single Flink
release. From my point of view, there should be a single parent module
that contains *independent modules* for each extension/library (there
should be no "cross dependencies" between the modules and each module
can specify the flink dependencies it needs by itself). This make is
most flexible. And if a library works on an old release, it might just
stay there as is. If a library changes (due to Flink changes), it might
just be contained multiple times for different Flink releases.

Each module should provide a short doc (README) that shows how to use an
integrate it with Flink. Thus, the responsibility goes to the
contributor to maintain the library. If it breaks and is not maintained
any further, we can simple remove it.

I agree, that the community might not be able to maintain those
extension/libraries right now. I would put the responsibility (more or
less completely) on the contributor and delete project that do not fix
any more.

@Vasia: a link to a library could be included in the README. If anybody
only wants to share a library but not contribute code, the parent README
could contain a list of additional links.


On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
> How about, for now, we simply create a page where we gather links/short
> descriptions of all these contributions
> and let the maintenance and dependency management to the tool/library
> creators?
> This way we will at least have these contributions in one place and link to
> them somewhere from the website.
> -Vasia.
> On 8 October 2015 at 12:06, Maximilian Michels <mxm@apache.org> wrote:
>> Hi Matthias,
>> Thanks for bringing up this idea. Actually, it has been discussed a
>> couple of times on the mailing list whether we should have a central
>> place for third-party extensions/contributions/libraries. This could
>> either be something package-based or, like you proposed, another
>> repository.
>> An external place for contributions raises a couple of questions
>> - Which version should the external contributions be based on?
>> - How do we make sure, the extensions are continuously updated?
>> (dedicated maintainers or automatic compatibility checks)
>> - How do we easily plug-in the external modules into Flink?
>> In the long term, we really need a solution for these questions. The
>> code base of Flink is growing and more and more packages go to
>> flink-contrib/flink-staging. I would find something packaged-based
>> better than a repository. Quite frankly, momentarily, I think
>> developing such a plugin system is out of scope for most Flink
>> developers. At the current pace of Flink development, collecting these
>> contributions externally without properly maintaining them, doesn't
>> make much sense to me.
>> Cheers,
>> Max
>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <mjsax@apache.org> wrote:
>>> Hi,
>>> many people are building quite exiting stuff on top of Flink. It is hard
>>> to keep an good overview on what stuff is available and what not. What
>>> do you think about starting a second git repository "flink-external"
>>> that collects all those code?
>>> The ideas would be to collect stuff in a central point, such that people
>>> can access it easily and get an overview what is already available (this
>>> might also avoid duplicate development). It might also be a good point
>>> to show common patterns. In order to collect as much as possible, the
>>> contributing requirement (with respect to testing etc) could be lower
>>> than for Flink itself.
>>> For example, I recently started a small flink-clojure module with a
>>> simple word-count example to answer a question on SO. Including this in
>>> Flink would not be appropriate. However, for a flink-external repro it
>>> might be nice to have.
>>> What do you think about it?
>>> -Matthias

View raw message