flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: [DISCUSS] flink-external
Date Fri, 09 Oct 2015 13:29:42 GMT
I was not sure what we should add and was hoping for input from the

I am aware of the following projects we might want to add:

  - Zeppelin
  - Mahout
  - Cascading (dataartisan repo)
  - BigPetStore
  - Gradoop


On 10/09/2015 03:07 PM, Maximilian Michels wrote:
> Cool. Right now the list is empty. Do you already have a list you
> could include in the upcoming pull request? :)
> On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <mjsax@apache.org> wrote:
>> Hi,
>> I just started this. Please see
>> https://github.com/mjsax/flink-web/tree/flink-external-page
>> I think, it is the best way to extend the "Downloads" page. I would also
>> add a link to this on the main page's "Getting Started" section.
>> As a first try, I started like this:
>>> Third party packages
>>> This is a list of third party packages (ie, libraries, system extensions, or
examples) build for Flink. The Flink community only collects links to those packages but does
not maintain them. Thus, they do not belong to the Apache Flink project and the community
cannot give any support for them.
>>> Package Name
>>> Available for Flink 0.8.x and 0.9.x
>>> Short description
>>> Please let us know, if we missed to list your package. Be aware, that we might
remove listed packages without notice.
>> Can you please give me some input, what projects I should add initially?
>> -Matthias
>> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>>> IMHO we can do that. There should be a disclaimer that the third party
>>> software is not officially supported.
>>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mjsax@apache.org> wrote:
>>>> Should we add a new page at Flink project web page?
>>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
>>>>> party software using Flink should be enough for now; of course,
>>>>> outside the Apache realm.
>>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanpark@apache.org>
>>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site
like Spark Packages [1] would be helpful to manage external contribution.
>>>>>> [1] http://spark-packages.org
>>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mjsax@apache.org>
>>>>>>> Thanks for the feedback.
>>>>>>> I think, the repository does not need to build on a single Flink
>>>>>>> release. From my point of view, there should be a single parent
>>>>>>> that contains *independent modules* for each extension/library
>>>>>>> should be no "cross dependencies" between the modules and each
>>>>>>> can specify the flink dependencies it needs by itself). This
make is
>>>>>>> most flexible. And if a library works on an old release, it might
>>>>>>> stay there as is. If a library changes (due to Flink changes),
it might
>>>>>>> just be contained multiple times for different Flink releases.
>>>>>>> Each module should provide a short doc (README) that shows how
to use an
>>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>>>> contributor to maintain the library. If it breaks and is not
>>>>>>> any further, we can simple remove it.
>>>>>>> I agree, that the community might not be able to maintain those
>>>>>>> extension/libraries right now. I would put the responsibility
(more or
>>>>>>> less completely) on the contributor and delete project that do
not fix
>>>>>>> any more.
>>>>>>> @Vasia: a link to a library could be included in the README.
If anybody
>>>>>>> only wants to share a library but not contribute code, the parent
>>>>>>> could contain a list of additional links.
>>>>>>> -Matthias
>>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>>> How about, for now, we simply create a page where we gather
>>>>>>>> descriptions of all these contributions
>>>>>>>> and let the maintenance and dependency management to the
>>>>>>>> creators?
>>>>>>>> This way we will at least have these contributions in one
place and link to
>>>>>>>> them somewhere from the website.
>>>>>>>> -Vasia.
>>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <mxm@apache.org>
>>>>>>>>> Hi Matthias,
>>>>>>>>> Thanks for bringing up this idea. Actually, it has been
discussed a
>>>>>>>>> couple of times on the mailing list whether we should
have a central
>>>>>>>>> place for third-party extensions/contributions/libraries.
This could
>>>>>>>>> either be something package-based or, like you proposed,
>>>>>>>>> repository.
>>>>>>>>> An external place for contributions raises a couple of
>>>>>>>>> - Which version should the external contributions be
based on?
>>>>>>>>> - How do we make sure, the extensions are continuously
>>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>>>> - How do we easily plug-in the external modules into
>>>>>>>>> In the long term, we really need a solution for these
questions. The
>>>>>>>>> code base of Flink is growing and more and more packages
go to
>>>>>>>>> flink-contrib/flink-staging. I would find something packaged-based
>>>>>>>>> better than a repository. Quite frankly, momentarily,
I think
>>>>>>>>> developing such a plugin system is out of scope for most
>>>>>>>>> developers. At the current pace of Flink development,
collecting these
>>>>>>>>> contributions externally without properly maintaining
them, doesn't
>>>>>>>>> make much sense to me.
>>>>>>>>> Cheers,
>>>>>>>>> Max
>>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <mjsax@apache.org>
>>>>>>>>>> Hi,
>>>>>>>>>> many people are building quite exiting stuff on top
of Flink. It is hard
>>>>>>>>>> to keep an good overview on what stuff is available
and what not. What
>>>>>>>>>> do you think about starting a second git repository
>>>>>>>>>> that collects all those code?
>>>>>>>>>> The ideas would be to collect stuff in a central
point, such that people
>>>>>>>>>> can access it easily and get an overview what is
already available (this
>>>>>>>>>> might also avoid duplicate development). It might
also be a good point
>>>>>>>>>> to show common patterns. In order to collect as much
as possible, the
>>>>>>>>>> contributing requirement (with respect to testing
etc) could be lower
>>>>>>>>>> than for Flink itself.
>>>>>>>>>> For example, I recently started a small flink-clojure
module with a
>>>>>>>>>> simple word-count example to answer a question on
SO. Including this in
>>>>>>>>>> Flink would not be appropriate. However, for a flink-external
repro it
>>>>>>>>>> might be nice to have.
>>>>>>>>>> What do you think about it?
>>>>>>>>>> -Matthias
>>>>>> Regards,
>>>>>> Chiwan Park

View raw message