flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: [DISCUSS] flink-external
Date Thu, 08 Oct 2015 14:03:48 GMT
IMHO we can do that. There should be a disclaimer that the third party
software is not officially supported.

On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mjsax@apache.org> wrote:
> Should we add a new page at Flink project web page?
>
> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>> +1 for your pragmatic approach, Vasia. A simple collection of third
>> party software using Flink should be enough for now; of course,
>> outside the Apache realm.
>>
>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanpark@apache.org> wrote:
>>> +1 for Vasia’s suggestion. From a long-term perspective, the site like Spark
Packages [1] would be helpful to manage external contribution.
>>>
>>> [1] http://spark-packages.org
>>>
>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mjsax@apache.org> wrote:
>>>>
>>>> Thanks for the feedback.
>>>>
>>>> I think, the repository does not need to build on a single Flink
>>>> release. From my point of view, there should be a single parent module
>>>> that contains *independent modules* for each extension/library (there
>>>> should be no "cross dependencies" between the modules and each module
>>>> can specify the flink dependencies it needs by itself). This make is
>>>> most flexible. And if a library works on an old release, it might just
>>>> stay there as is. If a library changes (due to Flink changes), it might
>>>> just be contained multiple times for different Flink releases.
>>>>
>>>> Each module should provide a short doc (README) that shows how to use an
>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>> contributor to maintain the library. If it breaks and is not maintained
>>>> any further, we can simple remove it.
>>>>
>>>> I agree, that the community might not be able to maintain those
>>>> extension/libraries right now. I would put the responsibility (more or
>>>> less completely) on the contributor and delete project that do not fix
>>>> any more.
>>>>
>>>> @Vasia: a link to a library could be included in the README. If anybody
>>>> only wants to share a library but not contribute code, the parent README
>>>> could contain a list of additional links.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>> How about, for now, we simply create a page where we gather links/short
>>>>> descriptions of all these contributions
>>>>> and let the maintenance and dependency management to the tool/library
>>>>> creators?
>>>>> This way we will at least have these contributions in one place and link
to
>>>>> them somewhere from the website.
>>>>>
>>>>> -Vasia.
>>>>>
>>>>> On 8 October 2015 at 12:06, Maximilian Michels <mxm@apache.org>
wrote:
>>>>>
>>>>>> Hi Matthias,
>>>>>>
>>>>>> Thanks for bringing up this idea. Actually, it has been discussed
a
>>>>>> couple of times on the mailing list whether we should have a central
>>>>>> place for third-party extensions/contributions/libraries. This could
>>>>>> either be something package-based or, like you proposed, another
>>>>>> repository.
>>>>>>
>>>>>> An external place for contributions raises a couple of questions
>>>>>>
>>>>>> - Which version should the external contributions be based on?
>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>
>>>>>> In the long term, we really need a solution for these questions.
The
>>>>>> code base of Flink is growing and more and more packages go to
>>>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>> developers. At the current pace of Flink development, collecting
these
>>>>>> contributions externally without properly maintaining them, doesn't
>>>>>> make much sense to me.
>>>>>>
>>>>>> Cheers,
>>>>>> Max
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <mjsax@apache.org>
wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> many people are building quite exiting stuff on top of Flink.
It is hard
>>>>>>> to keep an good overview on what stuff is available and what
not. What
>>>>>>> do you think about starting a second git repository "flink-external"
>>>>>>> that collects all those code?
>>>>>>>
>>>>>>> The ideas would be to collect stuff in a central point, such
that people
>>>>>>> can access it easily and get an overview what is already available
(this
>>>>>>> might also avoid duplicate development). It might also be a good
point
>>>>>>> to show common patterns. In order to collect as much as possible,
the
>>>>>>> contributing requirement (with respect to testing etc) could
be lower
>>>>>>> than for Flink itself.
>>>>>>>
>>>>>>> For example, I recently started a small flink-clojure module
with a
>>>>>>> simple word-count example to answer a question on SO. Including
this in
>>>>>>> Flink would not be appropriate. However, for a flink-external
repro it
>>>>>>> might be nice to have.
>>>>>>>
>>>>>>> What do you think about it?
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> Regards,
>>> Chiwan Park
>>>
>>>
>>>
>

Mime
View raw message