flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: [DISCUSS] flink-external
Date Fri, 09 Oct 2015 13:34:41 GMT
Yes, Community is a better place. You can also add the Dataflow Runner
https://github.com/dataArtisans/flink-dataflow.

On Fri, Oct 9, 2015 at 3:32 PM, Vasiliki Kalavri
<vasilikikalavri@gmail.com> wrote:
> Thank you Matthias!
>
> I'm not sure where the "Downloads" section is the right place for this.
> I would actually put it under "Community", with a header "External
> Contributions" or something like this, but I'm not feeling strong about
> this :)
>
> -Vasia.
>
>
> On 9 October 2015 at 15:29, Matthias J. Sax <mjsax@apache.org> wrote:
>
>> I was not sure what we should add and was hoping for input from the
>> community.
>>
>> I am aware of the following projects we might want to add:
>>
>>   - Zeppelin
>>   - SAMOA
>>   - Mahout
>>   - Cascading (dataartisan repo)
>>   - BigPetStore
>>   - Gradoop
>>
>>
>> -Matthias
>>
>>
>>
>> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
>> > Cool. Right now the list is empty. Do you already have a list you
>> > could include in the upcoming pull request? :)
>> >
>> > On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <mjsax@apache.org>
>> wrote:
>> >> Hi,
>> >>
>> >> I just started this. Please see
>> >> https://github.com/mjsax/flink-web/tree/flink-external-page
>> >>
>> >> I think, it is the best way to extend the "Downloads" page. I would also
>> >> add a link to this on the main page's "Getting Started" section.
>> >>
>> >> As a first try, I started like this:
>> >>> Third party packages
>> >>>
>> >>> This is a list of third party packages (ie, libraries, system
>> extensions, or examples) build for Flink. The Flink community only collects
>> links to those packages but does not maintain them. Thus, they do not
>> belong to the Apache Flink project and the community cannot give any
>> support for them.
>> >>> Package Name
>> >>>
>> >>> Available for Flink 0.8.x and 0.9.x
>> >>>
>> >>> Short description
>> >>>
>> >>> Please let us know, if we missed to list your package. Be aware, that
>> we might remove listed packages without notice.
>> >>
>> >> Can you please give me some input, what projects I should add initially?
>> >>
>> >>
>> >> -Matthias
>> >>
>> >>
>> >> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>> >>> IMHO we can do that. There should be a disclaimer that the third party
>> >>> software is not officially supported.
>> >>>
>> >>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mjsax@apache.org>
>> wrote:
>> >>>> Should we add a new page at Flink project web page?
>> >>>>
>> >>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>> >>>>> +1 for your pragmatic approach, Vasia. A simple collection of
third
>> >>>>> party software using Flink should be enough for now; of course,
>> >>>>> outside the Apache realm.
>> >>>>>
>> >>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanpark@apache.org>
>> wrote:
>> >>>>>> +1 for Vasia’s suggestion. From a long-term perspective,
the site
>> like Spark Packages [1] would be helpful to manage external contribution.
>> >>>>>>
>> >>>>>> [1] http://spark-packages.org
>> >>>>>>
>> >>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mjsax@apache.org>
>> wrote:
>> >>>>>>>
>> >>>>>>> Thanks for the feedback.
>> >>>>>>>
>> >>>>>>> I think, the repository does not need to build on a
single Flink
>> >>>>>>> release. From my point of view, there should be a single
parent
>> module
>> >>>>>>> that contains *independent modules* for each extension/library
>> (there
>> >>>>>>> should be no "cross dependencies" between the modules
and each
>> module
>> >>>>>>> can specify the flink dependencies it needs by itself).
This make
>> is
>> >>>>>>> most flexible. And if a library works on an old release,
it might
>> just
>> >>>>>>> stay there as is. If a library changes (due to Flink
changes), it
>> might
>> >>>>>>> just be contained multiple times for different Flink
releases.
>> >>>>>>>
>> >>>>>>> Each module should provide a short doc (README) that
shows how to
>> use an
>> >>>>>>> integrate it with Flink. Thus, the responsibility goes
to the
>> >>>>>>> contributor to maintain the library. If it breaks and
is not
>> maintained
>> >>>>>>> any further, we can simple remove it.
>> >>>>>>>
>> >>>>>>> I agree, that the community might not be able to maintain
those
>> >>>>>>> extension/libraries right now. I would put the responsibility
>> (more or
>> >>>>>>> less completely) on the contributor and delete project
that do not
>> fix
>> >>>>>>> any more.
>> >>>>>>>
>> >>>>>>> @Vasia: a link to a library could be included in the
README. If
>> anybody
>> >>>>>>> only wants to share a library but not contribute code,
the parent
>> README
>> >>>>>>> could contain a list of additional links.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -Matthias
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>> >>>>>>>> How about, for now, we simply create a page where
we gather
>> links/short
>> >>>>>>>> descriptions of all these contributions
>> >>>>>>>> and let the maintenance and dependency management
to the
>> tool/library
>> >>>>>>>> creators?
>> >>>>>>>> This way we will at least have these contributions
in one place
>> and link to
>> >>>>>>>> them somewhere from the website.
>> >>>>>>>>
>> >>>>>>>> -Vasia.
>> >>>>>>>>
>> >>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <mxm@apache.org>
>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hi Matthias,
>> >>>>>>>>>
>> >>>>>>>>> Thanks for bringing up this idea. Actually,
it has been
>> discussed a
>> >>>>>>>>> couple of times on the mailing list whether
we should have a
>> central
>> >>>>>>>>> place for third-party extensions/contributions/libraries.
This
>> could
>> >>>>>>>>> either be something package-based or, like you
proposed, another
>> >>>>>>>>> repository.
>> >>>>>>>>>
>> >>>>>>>>> An external place for contributions raises a
couple of questions
>> >>>>>>>>>
>> >>>>>>>>> - Which version should the external contributions
be based on?
>> >>>>>>>>> - How do we make sure, the extensions are continuously
updated?
>> >>>>>>>>> (dedicated maintainers or automatic compatibility
checks)
>> >>>>>>>>> - How do we easily plug-in the external modules
into Flink?
>> >>>>>>>>>
>> >>>>>>>>> In the long term, we really need a solution
for these questions.
>> The
>> >>>>>>>>> code base of Flink is growing and more and more
packages go to
>> >>>>>>>>> flink-contrib/flink-staging. I would find something
>> packaged-based
>> >>>>>>>>> better than a repository. Quite frankly, momentarily,
I think
>> >>>>>>>>> developing such a plugin system is out of scope
for most Flink
>> >>>>>>>>> developers. At the current pace of Flink development,
collecting
>> these
>> >>>>>>>>> contributions externally without properly maintaining
them,
>> doesn't
>> >>>>>>>>> make much sense to me.
>> >>>>>>>>>
>> >>>>>>>>> Cheers,
>> >>>>>>>>> Max
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J.
Sax <
>> mjsax@apache.org> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Hi,
>> >>>>>>>>>>
>> >>>>>>>>>> many people are building quite exiting stuff
on top of Flink.
>> It is hard
>> >>>>>>>>>> to keep an good overview on what stuff is
available and what
>> not. What
>> >>>>>>>>>> do you think about starting a second git
repository
>> "flink-external"
>> >>>>>>>>>> that collects all those code?
>> >>>>>>>>>>
>> >>>>>>>>>> The ideas would be to collect stuff in a
central point, such
>> that people
>> >>>>>>>>>> can access it easily and get an overview
what is already
>> available (this
>> >>>>>>>>>> might also avoid duplicate development).
It might also be a
>> good point
>> >>>>>>>>>> to show common patterns. In order to collect
as much as
>> possible, the
>> >>>>>>>>>> contributing requirement (with respect to
testing etc) could be
>> lower
>> >>>>>>>>>> than for Flink itself.
>> >>>>>>>>>>
>> >>>>>>>>>> For example, I recently started a small
flink-clojure module
>> with a
>> >>>>>>>>>> simple word-count example to answer a question
on SO. Including
>> this in
>> >>>>>>>>>> Flink would not be appropriate. However,
for a flink-external
>> repro it
>> >>>>>>>>>> might be nice to have.
>> >>>>>>>>>>
>> >>>>>>>>>> What do you think about it?
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> -Matthias
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> Regards,
>> >>>>>> Chiwan Park
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>
>>
>>

Mime
View raw message