flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: [DISCUSS] flink-external
Date Thu, 08 Oct 2015 10:56:25 GMT
+1 for your pragmatic approach, Vasia. A simple collection of third
party software using Flink should be enough for now; of course,
outside the Apache realm.

On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanpark@apache.org> wrote:
> +1 for Vasia’s suggestion. From a long-term perspective, the site like Spark Packages
[1] would be helpful to manage external contribution.
>
> [1] http://spark-packages.org
>
>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mjsax@apache.org> wrote:
>>
>> Thanks for the feedback.
>>
>> I think, the repository does not need to build on a single Flink
>> release. From my point of view, there should be a single parent module
>> that contains *independent modules* for each extension/library (there
>> should be no "cross dependencies" between the modules and each module
>> can specify the flink dependencies it needs by itself). This make is
>> most flexible. And if a library works on an old release, it might just
>> stay there as is. If a library changes (due to Flink changes), it might
>> just be contained multiple times for different Flink releases.
>>
>> Each module should provide a short doc (README) that shows how to use an
>> integrate it with Flink. Thus, the responsibility goes to the
>> contributor to maintain the library. If it breaks and is not maintained
>> any further, we can simple remove it.
>>
>> I agree, that the community might not be able to maintain those
>> extension/libraries right now. I would put the responsibility (more or
>> less completely) on the contributor and delete project that do not fix
>> any more.
>>
>> @Vasia: a link to a library could be included in the README. If anybody
>> only wants to share a library but not contribute code, the parent README
>> could contain a list of additional links.
>>
>>
>> -Matthias
>>
>>
>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>> How about, for now, we simply create a page where we gather links/short
>>> descriptions of all these contributions
>>> and let the maintenance and dependency management to the tool/library
>>> creators?
>>> This way we will at least have these contributions in one place and link to
>>> them somewhere from the website.
>>>
>>> -Vasia.
>>>
>>> On 8 October 2015 at 12:06, Maximilian Michels <mxm@apache.org> wrote:
>>>
>>>> Hi Matthias,
>>>>
>>>> Thanks for bringing up this idea. Actually, it has been discussed a
>>>> couple of times on the mailing list whether we should have a central
>>>> place for third-party extensions/contributions/libraries. This could
>>>> either be something package-based or, like you proposed, another
>>>> repository.
>>>>
>>>> An external place for contributions raises a couple of questions
>>>>
>>>> - Which version should the external contributions be based on?
>>>> - How do we make sure, the extensions are continuously updated?
>>>> (dedicated maintainers or automatic compatibility checks)
>>>> - How do we easily plug-in the external modules into Flink?
>>>>
>>>> In the long term, we really need a solution for these questions. The
>>>> code base of Flink is growing and more and more packages go to
>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>> better than a repository. Quite frankly, momentarily, I think
>>>> developing such a plugin system is out of scope for most Flink
>>>> developers. At the current pace of Flink development, collecting these
>>>> contributions externally without properly maintaining them, doesn't
>>>> make much sense to me.
>>>>
>>>> Cheers,
>>>> Max
>>>>
>>>>
>>>>
>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <mjsax@apache.org>
wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> many people are building quite exiting stuff on top of Flink. It is hard
>>>>> to keep an good overview on what stuff is available and what not. What
>>>>> do you think about starting a second git repository "flink-external"
>>>>> that collects all those code?
>>>>>
>>>>> The ideas would be to collect stuff in a central point, such that people
>>>>> can access it easily and get an overview what is already available (this
>>>>> might also avoid duplicate development). It might also be a good point
>>>>> to show common patterns. In order to collect as much as possible, the
>>>>> contributing requirement (with respect to testing etc) could be lower
>>>>> than for Flink itself.
>>>>>
>>>>> For example, I recently started a small flink-clojure module with a
>>>>> simple word-count example to answer a question on SO. Including this
in
>>>>> Flink would not be appropriate. However, for a flink-external repro it
>>>>> might be nice to have.
>>>>>
>>>>> What do you think about it?
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>
>>>
>>
>
>
>
> Regards,
> Chiwan Park
>
>
>

Mime
View raw message