Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <5617B31C.6040301@apache.org>
References: <5614E8FA.70208@apache.org>
 <CAGco--bpgz1XqrcHZdW-kjgn0JobS1ExqFCNfZY3mwJ10KhgrA@mail.gmail.com>
 <CAJZ2dcWL17boidAQxhMa8hSM9M2QWx9cf2YoSmaGLUDSKYD_tw@mail.gmail.com>
 <56164542.60209@apache.org> <2F276189-2C43-4A63-AFF0-EEF2B699F69B@apache.org>
 <CAGco--aHk1_vxzMNcUt8k2ERTsDn3Ex=m256RP=ti=DdKNOuow@mail.gmail.com>
 <561660D5.6040208@apache.org>
 <CAGco--ZHaVjy9PwEBjHF1f7COTaFutTouqiKJpEt0DdXD6ranQ@mail.gmail.com>
 <5617B31C.6040301@apache.org>
From: Maximilian Michels <mxm@apache.org>
Date: Fri, 9 Oct 2015 15:07:02 +0200
Message-ID: 
 <CAGco--bV6WmfTeunK+9kvbkoN3yMA2+oOwp7DSiXq8x8gr2aJQ@mail.gmail.com>
Subject: Re: [DISCUSS] flink-external
To: dev@flink.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Cool. Right now the list is empty. Do you already have a list you
could include in the upcoming pull request? :)

On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <mjsax@apache.org> wrote:
> Hi,
>
> I just started this. Please see
> https://github.com/mjsax/flink-web/tree/flink-external-page
>
> I think, it is the best way to extend the "Downloads" page. I would also
> add a link to this on the main page's "Getting Started" section.
>
> As a first try, I started like this:
>> Third party packages
>>
>> This is a list of third party packages (ie, libraries, system extensions=
, or examples) build for Flink. The Flink community only collects links to =
those packages but does not maintain them. Thus, they do not belong to the =
Apache Flink project and the community cannot give any support for them.
>> Package Name
>>
>> Available for Flink 0.8.x and 0.9.x
>>
>> Short description
>>
>> Please let us know, if we missed to list your package. Be aware, that we=
 might remove listed packages without notice.
>
> Can you please give me some input, what projects I should add initially?
>
>
> -Matthias
>
>
> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>> IMHO we can do that. There should be a disclaimer that the third party
>> software is not officially supported.
>>
>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mjsax@apache.org> wrote=
:
>>> Should we add a new page at Flink project web page?
>>>
>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>>>> party software using Flink should be enough for now; of course,
>>>> outside the Apache realm.
>>>>
>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanpark@apache.org> w=
rote:
>>>>> +1 for Vasia=E2=80=99s suggestion. From a long-term perspective, the =
site like Spark Packages [1] would be helpful to manage external contributi=
on.
>>>>>
>>>>> [1] http://spark-packages.org
>>>>>
>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mjsax@apache.org> wrot=
e:
>>>>>>
>>>>>> Thanks for the feedback.
>>>>>>
>>>>>> I think, the repository does not need to build on a single Flink
>>>>>> release. From my point of view, there should be a single parent modu=
le
>>>>>> that contains *independent modules* for each extension/library (ther=
e
>>>>>> should be no "cross dependencies" between the modules and each modul=
e
>>>>>> can specify the flink dependencies it needs by itself). This make is
>>>>>> most flexible. And if a library works on an old release, it might ju=
st
>>>>>> stay there as is. If a library changes (due to Flink changes), it mi=
ght
>>>>>> just be contained multiple times for different Flink releases.
>>>>>>
>>>>>> Each module should provide a short doc (README) that shows how to us=
e an
>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>>> contributor to maintain the library. If it breaks and is not maintai=
ned
>>>>>> any further, we can simple remove it.
>>>>>>
>>>>>> I agree, that the community might not be able to maintain those
>>>>>> extension/libraries right now. I would put the responsibility (more =
or
>>>>>> less completely) on the contributor and delete project that do not f=
ix
>>>>>> any more.
>>>>>>
>>>>>> @Vasia: a link to a library could be included in the README. If anyb=
ody
>>>>>> only wants to share a library but not contribute code, the parent RE=
ADME
>>>>>> could contain a list of additional links.
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>> How about, for now, we simply create a page where we gather links/s=
hort
>>>>>>> descriptions of all these contributions
>>>>>>> and let the maintenance and dependency management to the tool/libra=
ry
>>>>>>> creators?
>>>>>>> This way we will at least have these contributions in one place and=
 link to
>>>>>>> them somewhere from the website.
>>>>>>>
>>>>>>> -Vasia.
>>>>>>>
>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <mxm@apache.org> wro=
te:
>>>>>>>
>>>>>>>> Hi Matthias,
>>>>>>>>
>>>>>>>> Thanks for bringing up this idea. Actually, it has been discussed =
a
>>>>>>>> couple of times on the mailing list whether we should have a centr=
al
>>>>>>>> place for third-party extensions/contributions/libraries. This cou=
ld
>>>>>>>> either be something package-based or, like you proposed, another
>>>>>>>> repository.
>>>>>>>>
>>>>>>>> An external place for contributions raises a couple of questions
>>>>>>>>
>>>>>>>> - Which version should the external contributions be based on?
>>>>>>>> - How do we make sure, the extensions are continuously updated?
>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>>>
>>>>>>>> In the long term, we really need a solution for these questions. T=
he
>>>>>>>> code base of Flink is growing and more and more packages go to
>>>>>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>>>> developers. At the current pace of Flink development, collecting t=
hese
>>>>>>>> contributions externally without properly maintaining them, doesn'=
t
>>>>>>>> make much sense to me.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Max
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <mjsax@apache.org=
> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> many people are building quite exiting stuff on top of Flink. It =
is hard
>>>>>>>>> to keep an good overview on what stuff is available and what not.=
 What
>>>>>>>>> do you think about starting a second git repository "flink-extern=
al"
>>>>>>>>> that collects all those code?
>>>>>>>>>
>>>>>>>>> The ideas would be to collect stuff in a central point, such that=
 people
>>>>>>>>> can access it easily and get an overview what is already availabl=
e (this
>>>>>>>>> might also avoid duplicate development). It might also be a good =
point
>>>>>>>>> to show common patterns. In order to collect as much as possible,=
 the
>>>>>>>>> contributing requirement (with respect to testing etc) could be l=
ower
>>>>>>>>> than for Flink itself.
>>>>>>>>>
>>>>>>>>> For example, I recently started a small flink-clojure module with=
 a
>>>>>>>>> simple word-count example to answer a question on SO. Including t=
his in
>>>>>>>>> Flink would not be appropriate. However, for a flink-external rep=
ro it
>>>>>>>>> might be nice to have.
>>>>>>>>>
>>>>>>>>> What do you think about it?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Chiwan Park
>>>>>
>>>>>
>>>>>
>>>
>