flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: [DISCUSS] flink-external
Date Mon, 02 Nov 2015 13:48:46 GMT
What is the status here: https://github.com/apache/flink-web/pull/12

Please give feedback.

On 10/28/2015 11:02 AM, Maximilian Michels wrote:
> Thanks Matthias! I made a comment. Please open a pull request.
> 
> On Tue, Oct 27, 2015 at 10:37 PM, Matthias J. Sax <mjsax@apache.org> wrote:
>> Just updated this. Improved the layout and added FastR project.
>>
>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>
>> -Matthias
>>
>> On 10/27/2015 03:25 AM, Matthias J. Sax wrote:
>>> Hi,
>>>
>>> I updated the flink-external section on the Flink Web-Page:
>>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>>
>>> The section is now located in "Contribute" page. The layout needs some
>>> refinement though... Some Project are "previews", ie, Flink support was
>>> announced but there is not information on the according project web
>>> pages. We might want to reach out to those people to see if we should
>>> include those project already or just add them later on.
>>>
>>> Please give feedback.
>>>
>>>
>>> -Matthias
>>>
>>>
>>>
>>>
>>> On 10/09/2015 03:34 PM, Maximilian Michels wrote:
>>>> Yes, Community is a better place. You can also add the Dataflow Runner
>>>> https://github.com/dataArtisans/flink-dataflow.
>>>>
>>>> On Fri, Oct 9, 2015 at 3:32 PM, Vasiliki Kalavri
>>>> <vasilikikalavri@gmail.com> wrote:
>>>>> Thank you Matthias!
>>>>>
>>>>> I'm not sure where the "Downloads" section is the right place for this.
>>>>> I would actually put it under "Community", with a header "External
>>>>> Contributions" or something like this, but I'm not feeling strong about
>>>>> this :)
>>>>>
>>>>> -Vasia.
>>>>>
>>>>>
>>>>> On 9 October 2015 at 15:29, Matthias J. Sax <mjsax@apache.org>
wrote:
>>>>>
>>>>>> I was not sure what we should add and was hoping for input from the
>>>>>> community.
>>>>>>
>>>>>> I am aware of the following projects we might want to add:
>>>>>>
>>>>>>   - Zeppelin
>>>>>>   - SAMOA
>>>>>>   - Mahout
>>>>>>   - Cascading (dataartisan repo)
>>>>>>   - BigPetStore
>>>>>>   - Gradoop
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
>>>>>>> Cool. Right now the list is empty. Do you already have a list
you
>>>>>>> could include in the upcoming pull request? :)
>>>>>>>
>>>>>>> On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <mjsax@apache.org>
>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I just started this. Please see
>>>>>>>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>>>>>>>
>>>>>>>> I think, it is the best way to extend the "Downloads" page.
I would also
>>>>>>>> add a link to this on the main page's "Getting Started" section.
>>>>>>>>
>>>>>>>> As a first try, I started like this:
>>>>>>>>> Third party packages
>>>>>>>>>
>>>>>>>>> This is a list of third party packages (ie, libraries,
system
>>>>>> extensions, or examples) build for Flink. The Flink community only
collects
>>>>>> links to those packages but does not maintain them. Thus, they do
not
>>>>>> belong to the Apache Flink project and the community cannot give
any
>>>>>> support for them.
>>>>>>>>> Package Name
>>>>>>>>>
>>>>>>>>> Available for Flink 0.8.x and 0.9.x
>>>>>>>>>
>>>>>>>>> Short description
>>>>>>>>>
>>>>>>>>> Please let us know, if we missed to list your package.
Be aware, that
>>>>>> we might remove listed packages without notice.
>>>>>>>>
>>>>>>>> Can you please give me some input, what projects I should
add initially?
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>>>>>>>>> IMHO we can do that. There should be a disclaimer that
the third party
>>>>>>>>> software is not officially supported.
>>>>>>>>>
>>>>>>>>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mjsax@apache.org>
>>>>>> wrote:
>>>>>>>>>> Should we add a new page at Flink project web page?
>>>>>>>>>>
>>>>>>>>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>>>>>>>>> +1 for your pragmatic approach, Vasia. A simple
collection of third
>>>>>>>>>>> party software using Flink should be enough for
now; of course,
>>>>>>>>>>> outside the Apache realm.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park
<chiwanpark@apache.org>
>>>>>> wrote:
>>>>>>>>>>>> +1 for Vasia’s suggestion. From a long-term
perspective, the site
>>>>>> like Spark Packages [1] would be helpful to manage external contribution.
>>>>>>>>>>>>
>>>>>>>>>>>> [1] http://spark-packages.org
>>>>>>>>>>>>
>>>>>>>>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias
J. Sax <mjsax@apache.org>
>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think, the repository does not need
to build on a single Flink
>>>>>>>>>>>>> release. From my point of view, there
should be a single parent
>>>>>> module
>>>>>>>>>>>>> that contains *independent modules* for
each extension/library
>>>>>> (there
>>>>>>>>>>>>> should be no "cross dependencies" between
the modules and each
>>>>>> module
>>>>>>>>>>>>> can specify the flink dependencies it
needs by itself). This make
>>>>>> is
>>>>>>>>>>>>> most flexible. And if a library works
on an old release, it might
>>>>>> just
>>>>>>>>>>>>> stay there as is. If a library changes
(due to Flink changes), it
>>>>>> might
>>>>>>>>>>>>> just be contained multiple times for
different Flink releases.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Each module should provide a short doc
(README) that shows how to
>>>>>> use an
>>>>>>>>>>>>> integrate it with Flink. Thus, the responsibility
goes to the
>>>>>>>>>>>>> contributor to maintain the library.
If it breaks and is not
>>>>>> maintained
>>>>>>>>>>>>> any further, we can simple remove it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree, that the community might not
be able to maintain those
>>>>>>>>>>>>> extension/libraries right now. I would
put the responsibility
>>>>>> (more or
>>>>>>>>>>>>> less completely) on the contributor and
delete project that do not
>>>>>> fix
>>>>>>>>>>>>> any more.
>>>>>>>>>>>>>
>>>>>>>>>>>>> @Vasia: a link to a library could be
included in the README. If
>>>>>> anybody
>>>>>>>>>>>>> only wants to share a library but not
contribute code, the parent
>>>>>> README
>>>>>>>>>>>>> could contain a list of additional links.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri
wrote:
>>>>>>>>>>>>>> How about, for now, we simply create
a page where we gather
>>>>>> links/short
>>>>>>>>>>>>>> descriptions of all these contributions
>>>>>>>>>>>>>> and let the maintenance and dependency
management to the
>>>>>> tool/library
>>>>>>>>>>>>>> creators?
>>>>>>>>>>>>>> This way we will at least have these
contributions in one place
>>>>>> and link to
>>>>>>>>>>>>>> them somewhere from the website.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Vasia.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 8 October 2015 at 12:06, Maximilian
Michels <mxm@apache.org>
>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Matthias,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for bringing up this idea.
Actually, it has been
>>>>>> discussed a
>>>>>>>>>>>>>>> couple of times on the mailing
list whether we should have a
>>>>>> central
>>>>>>>>>>>>>>> place for third-party extensions/contributions/libraries.
This
>>>>>> could
>>>>>>>>>>>>>>> either be something package-based
or, like you proposed, another
>>>>>>>>>>>>>>> repository.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> An external place for contributions
raises a couple of questions
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Which version should the external
contributions be based on?
>>>>>>>>>>>>>>> - How do we make sure, the extensions
are continuously updated?
>>>>>>>>>>>>>>> (dedicated maintainers or automatic
compatibility checks)
>>>>>>>>>>>>>>> - How do we easily plug-in the
external modules into Flink?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In the long term, we really need
a solution for these questions.
>>>>>> The
>>>>>>>>>>>>>>> code base of Flink is growing
and more and more packages go to
>>>>>>>>>>>>>>> flink-contrib/flink-staging.
I would find something
>>>>>> packaged-based
>>>>>>>>>>>>>>> better than a repository. Quite
frankly, momentarily, I think
>>>>>>>>>>>>>>> developing such a plugin system
is out of scope for most Flink
>>>>>>>>>>>>>>> developers. At the current pace
of Flink development, collecting
>>>>>> these
>>>>>>>>>>>>>>> contributions externally without
properly maintaining them,
>>>>>> doesn't
>>>>>>>>>>>>>>> make much sense to me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Max
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 11:42
AM, Matthias J. Sax <
>>>>>> mjsax@apache.org> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> many people are building
quite exiting stuff on top of Flink.
>>>>>> It is hard
>>>>>>>>>>>>>>>> to keep an good overview
on what stuff is available and what
>>>>>> not. What
>>>>>>>>>>>>>>>> do you think about starting
a second git repository
>>>>>> "flink-external"
>>>>>>>>>>>>>>>> that collects all those code?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The ideas would be to collect
stuff in a central point, such
>>>>>> that people
>>>>>>>>>>>>>>>> can access it easily and
get an overview what is already
>>>>>> available (this
>>>>>>>>>>>>>>>> might also avoid duplicate
development). It might also be a
>>>>>> good point
>>>>>>>>>>>>>>>> to show common patterns.
In order to collect as much as
>>>>>> possible, the
>>>>>>>>>>>>>>>> contributing requirement
(with respect to testing etc) could be
>>>>>> lower
>>>>>>>>>>>>>>>> than for Flink itself.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For example, I recently started
a small flink-clojure module
>>>>>> with a
>>>>>>>>>>>>>>>> simple word-count example
to answer a question on SO. Including
>>>>>> this in
>>>>>>>>>>>>>>>> Flink would not be appropriate.
However, for a flink-external
>>>>>> repro it
>>>>>>>>>>>>>>>> might be nice to have.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What do you think about it?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Chiwan Park
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>
>>


Mime
View raw message