flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: [DISCUSS] flink-external
Date Tue, 27 Oct 2015 21:37:19 GMT
Just updated this. Improved the layout and added FastR project.

https://github.com/mjsax/flink-web/tree/flink-external-page

-Matthias

On 10/27/2015 03:25 AM, Matthias J. Sax wrote:
> Hi,
> 
> I updated the flink-external section on the Flink Web-Page:
> https://github.com/mjsax/flink-web/tree/flink-external-page
> 
> The section is now located in "Contribute" page. The layout needs some
> refinement though... Some Project are "previews", ie, Flink support was
> announced but there is not information on the according project web
> pages. We might want to reach out to those people to see if we should
> include those project already or just add them later on.
> 
> Please give feedback.
> 
> 
> -Matthias
> 
> 
> 
> 
> On 10/09/2015 03:34 PM, Maximilian Michels wrote:
>> Yes, Community is a better place. You can also add the Dataflow Runner
>> https://github.com/dataArtisans/flink-dataflow.
>>
>> On Fri, Oct 9, 2015 at 3:32 PM, Vasiliki Kalavri
>> <vasilikikalavri@gmail.com> wrote:
>>> Thank you Matthias!
>>>
>>> I'm not sure where the "Downloads" section is the right place for this.
>>> I would actually put it under "Community", with a header "External
>>> Contributions" or something like this, but I'm not feeling strong about
>>> this :)
>>>
>>> -Vasia.
>>>
>>>
>>> On 9 October 2015 at 15:29, Matthias J. Sax <mjsax@apache.org> wrote:
>>>
>>>> I was not sure what we should add and was hoping for input from the
>>>> community.
>>>>
>>>> I am aware of the following projects we might want to add:
>>>>
>>>>   - Zeppelin
>>>>   - SAMOA
>>>>   - Mahout
>>>>   - Cascading (dataartisan repo)
>>>>   - BigPetStore
>>>>   - Gradoop
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>>
>>>> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
>>>>> Cool. Right now the list is empty. Do you already have a list you
>>>>> could include in the upcoming pull request? :)
>>>>>
>>>>> On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <mjsax@apache.org>
>>>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I just started this. Please see
>>>>>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>>>>>
>>>>>> I think, it is the best way to extend the "Downloads" page. I would
also
>>>>>> add a link to this on the main page's "Getting Started" section.
>>>>>>
>>>>>> As a first try, I started like this:
>>>>>>> Third party packages
>>>>>>>
>>>>>>> This is a list of third party packages (ie, libraries, system
>>>> extensions, or examples) build for Flink. The Flink community only collects
>>>> links to those packages but does not maintain them. Thus, they do not
>>>> belong to the Apache Flink project and the community cannot give any
>>>> support for them.
>>>>>>> Package Name
>>>>>>>
>>>>>>> Available for Flink 0.8.x and 0.9.x
>>>>>>>
>>>>>>> Short description
>>>>>>>
>>>>>>> Please let us know, if we missed to list your package. Be aware,
that
>>>> we might remove listed packages without notice.
>>>>>>
>>>>>> Can you please give me some input, what projects I should add initially?
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>>>>>>> IMHO we can do that. There should be a disclaimer that the third
party
>>>>>>> software is not officially supported.
>>>>>>>
>>>>>>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mjsax@apache.org>
>>>> wrote:
>>>>>>>> Should we add a new page at Flink project web page?
>>>>>>>>
>>>>>>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>>>>>>> +1 for your pragmatic approach, Vasia. A simple collection
of third
>>>>>>>>> party software using Flink should be enough for now;
of course,
>>>>>>>>> outside the Apache realm.
>>>>>>>>>
>>>>>>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanpark@apache.org>
>>>> wrote:
>>>>>>>>>> +1 for Vasia’s suggestion. From a long-term perspective,
the site
>>>> like Spark Packages [1] would be helpful to manage external contribution.
>>>>>>>>>>
>>>>>>>>>> [1] http://spark-packages.org
>>>>>>>>>>
>>>>>>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax
<mjsax@apache.org>
>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>
>>>>>>>>>>> I think, the repository does not need to build
on a single Flink
>>>>>>>>>>> release. From my point of view, there should
be a single parent
>>>> module
>>>>>>>>>>> that contains *independent modules* for each
extension/library
>>>> (there
>>>>>>>>>>> should be no "cross dependencies" between the
modules and each
>>>> module
>>>>>>>>>>> can specify the flink dependencies it needs by
itself). This make
>>>> is
>>>>>>>>>>> most flexible. And if a library works on an old
release, it might
>>>> just
>>>>>>>>>>> stay there as is. If a library changes (due to
Flink changes), it
>>>> might
>>>>>>>>>>> just be contained multiple times for different
Flink releases.
>>>>>>>>>>>
>>>>>>>>>>> Each module should provide a short doc (README)
that shows how to
>>>> use an
>>>>>>>>>>> integrate it with Flink. Thus, the responsibility
goes to the
>>>>>>>>>>> contributor to maintain the library. If it breaks
and is not
>>>> maintained
>>>>>>>>>>> any further, we can simple remove it.
>>>>>>>>>>>
>>>>>>>>>>> I agree, that the community might not be able
to maintain those
>>>>>>>>>>> extension/libraries right now. I would put the
responsibility
>>>> (more or
>>>>>>>>>>> less completely) on the contributor and delete
project that do not
>>>> fix
>>>>>>>>>>> any more.
>>>>>>>>>>>
>>>>>>>>>>> @Vasia: a link to a library could be included
in the README. If
>>>> anybody
>>>>>>>>>>> only wants to share a library but not contribute
code, the parent
>>>> README
>>>>>>>>>>> could contain a list of additional links.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>>>>>>> How about, for now, we simply create a page
where we gather
>>>> links/short
>>>>>>>>>>>> descriptions of all these contributions
>>>>>>>>>>>> and let the maintenance and dependency management
to the
>>>> tool/library
>>>>>>>>>>>> creators?
>>>>>>>>>>>> This way we will at least have these contributions
in one place
>>>> and link to
>>>>>>>>>>>> them somewhere from the website.
>>>>>>>>>>>>
>>>>>>>>>>>> -Vasia.
>>>>>>>>>>>>
>>>>>>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels
<mxm@apache.org>
>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Matthias,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for bringing up this idea. Actually,
it has been
>>>> discussed a
>>>>>>>>>>>>> couple of times on the mailing list whether
we should have a
>>>> central
>>>>>>>>>>>>> place for third-party extensions/contributions/libraries.
This
>>>> could
>>>>>>>>>>>>> either be something package-based or,
like you proposed, another
>>>>>>>>>>>>> repository.
>>>>>>>>>>>>>
>>>>>>>>>>>>> An external place for contributions raises
a couple of questions
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Which version should the external contributions
be based on?
>>>>>>>>>>>>> - How do we make sure, the extensions
are continuously updated?
>>>>>>>>>>>>> (dedicated maintainers or automatic compatibility
checks)
>>>>>>>>>>>>> - How do we easily plug-in the external
modules into Flink?
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the long term, we really need a solution
for these questions.
>>>> The
>>>>>>>>>>>>> code base of Flink is growing and more
and more packages go to
>>>>>>>>>>>>> flink-contrib/flink-staging. I would
find something
>>>> packaged-based
>>>>>>>>>>>>> better than a repository. Quite frankly,
momentarily, I think
>>>>>>>>>>>>> developing such a plugin system is out
of scope for most Flink
>>>>>>>>>>>>> developers. At the current pace of Flink
development, collecting
>>>> these
>>>>>>>>>>>>> contributions externally without properly
maintaining them,
>>>> doesn't
>>>>>>>>>>>>> make much sense to me.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Max
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias
J. Sax <
>>>> mjsax@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> many people are building quite exiting
stuff on top of Flink.
>>>> It is hard
>>>>>>>>>>>>>> to keep an good overview on what
stuff is available and what
>>>> not. What
>>>>>>>>>>>>>> do you think about starting a second
git repository
>>>> "flink-external"
>>>>>>>>>>>>>> that collects all those code?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The ideas would be to collect stuff
in a central point, such
>>>> that people
>>>>>>>>>>>>>> can access it easily and get an overview
what is already
>>>> available (this
>>>>>>>>>>>>>> might also avoid duplicate development).
It might also be a
>>>> good point
>>>>>>>>>>>>>> to show common patterns. In order
to collect as much as
>>>> possible, the
>>>>>>>>>>>>>> contributing requirement (with respect
to testing etc) could be
>>>> lower
>>>>>>>>>>>>>> than for Flink itself.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For example, I recently started a
small flink-clojure module
>>>> with a
>>>>>>>>>>>>>> simple word-count example to answer
a question on SO. Including
>>>> this in
>>>>>>>>>>>>>> Flink would not be appropriate. However,
for a flink-external
>>>> repro it
>>>>>>>>>>>>>> might be nice to have.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you think about it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Chiwan Park
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>>
> 


Mime
View raw message