flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vasiliki Kalavri <vasilikikala...@gmail.com>
Subject Re: [DISCUSS] flink-external
Date Fri, 09 Oct 2015 13:32:22 GMT
Thank you Matthias!

I'm not sure where the "Downloads" section is the right place for this.
I would actually put it under "Community", with a header "External
Contributions" or something like this, but I'm not feeling strong about
this :)

-Vasia.


On 9 October 2015 at 15:29, Matthias J. Sax <mjsax@apache.org> wrote:

> I was not sure what we should add and was hoping for input from the
> community.
>
> I am aware of the following projects we might want to add:
>
>   - Zeppelin
>   - SAMOA
>   - Mahout
>   - Cascading (dataartisan repo)
>   - BigPetStore
>   - Gradoop
>
>
> -Matthias
>
>
>
> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
> > Cool. Right now the list is empty. Do you already have a list you
> > could include in the upcoming pull request? :)
> >
> > On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <mjsax@apache.org>
> wrote:
> >> Hi,
> >>
> >> I just started this. Please see
> >> https://github.com/mjsax/flink-web/tree/flink-external-page
> >>
> >> I think, it is the best way to extend the "Downloads" page. I would also
> >> add a link to this on the main page's "Getting Started" section.
> >>
> >> As a first try, I started like this:
> >>> Third party packages
> >>>
> >>> This is a list of third party packages (ie, libraries, system
> extensions, or examples) build for Flink. The Flink community only collects
> links to those packages but does not maintain them. Thus, they do not
> belong to the Apache Flink project and the community cannot give any
> support for them.
> >>> Package Name
> >>>
> >>> Available for Flink 0.8.x and 0.9.x
> >>>
> >>> Short description
> >>>
> >>> Please let us know, if we missed to list your package. Be aware, that
> we might remove listed packages without notice.
> >>
> >> Can you please give me some input, what projects I should add initially?
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
> >>> IMHO we can do that. There should be a disclaimer that the third party
> >>> software is not officially supported.
> >>>
> >>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mjsax@apache.org>
> wrote:
> >>>> Should we add a new page at Flink project web page?
> >>>>
> >>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
> >>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
> >>>>> party software using Flink should be enough for now; of course,
> >>>>> outside the Apache realm.
> >>>>>
> >>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanpark@apache.org>
> wrote:
> >>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the
site
> like Spark Packages [1] would be helpful to manage external contribution.
> >>>>>>
> >>>>>> [1] http://spark-packages.org
> >>>>>>
> >>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mjsax@apache.org>
> wrote:
> >>>>>>>
> >>>>>>> Thanks for the feedback.
> >>>>>>>
> >>>>>>> I think, the repository does not need to build on a single
Flink
> >>>>>>> release. From my point of view, there should be a single
parent
> module
> >>>>>>> that contains *independent modules* for each extension/library
> (there
> >>>>>>> should be no "cross dependencies" between the modules and
each
> module
> >>>>>>> can specify the flink dependencies it needs by itself).
This make
> is
> >>>>>>> most flexible. And if a library works on an old release,
it might
> just
> >>>>>>> stay there as is. If a library changes (due to Flink changes),
it
> might
> >>>>>>> just be contained multiple times for different Flink releases.
> >>>>>>>
> >>>>>>> Each module should provide a short doc (README) that shows
how to
> use an
> >>>>>>> integrate it with Flink. Thus, the responsibility goes to
the
> >>>>>>> contributor to maintain the library. If it breaks and is
not
> maintained
> >>>>>>> any further, we can simple remove it.
> >>>>>>>
> >>>>>>> I agree, that the community might not be able to maintain
those
> >>>>>>> extension/libraries right now. I would put the responsibility
> (more or
> >>>>>>> less completely) on the contributor and delete project that
do not
> fix
> >>>>>>> any more.
> >>>>>>>
> >>>>>>> @Vasia: a link to a library could be included in the README.
If
> anybody
> >>>>>>> only wants to share a library but not contribute code, the
parent
> README
> >>>>>>> could contain a list of additional links.
> >>>>>>>
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>>
> >>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
> >>>>>>>> How about, for now, we simply create a page where we
gather
> links/short
> >>>>>>>> descriptions of all these contributions
> >>>>>>>> and let the maintenance and dependency management to
the
> tool/library
> >>>>>>>> creators?
> >>>>>>>> This way we will at least have these contributions in
one place
> and link to
> >>>>>>>> them somewhere from the website.
> >>>>>>>>
> >>>>>>>> -Vasia.
> >>>>>>>>
> >>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <mxm@apache.org>
> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Matthias,
> >>>>>>>>>
> >>>>>>>>> Thanks for bringing up this idea. Actually, it has
been
> discussed a
> >>>>>>>>> couple of times on the mailing list whether we should
have a
> central
> >>>>>>>>> place for third-party extensions/contributions/libraries.
This
> could
> >>>>>>>>> either be something package-based or, like you proposed,
another
> >>>>>>>>> repository.
> >>>>>>>>>
> >>>>>>>>> An external place for contributions raises a couple
of questions
> >>>>>>>>>
> >>>>>>>>> - Which version should the external contributions
be based on?
> >>>>>>>>> - How do we make sure, the extensions are continuously
updated?
> >>>>>>>>> (dedicated maintainers or automatic compatibility
checks)
> >>>>>>>>> - How do we easily plug-in the external modules
into Flink?
> >>>>>>>>>
> >>>>>>>>> In the long term, we really need a solution for
these questions.
> The
> >>>>>>>>> code base of Flink is growing and more and more
packages go to
> >>>>>>>>> flink-contrib/flink-staging. I would find something
> packaged-based
> >>>>>>>>> better than a repository. Quite frankly, momentarily,
I think
> >>>>>>>>> developing such a plugin system is out of scope
for most Flink
> >>>>>>>>> developers. At the current pace of Flink development,
collecting
> these
> >>>>>>>>> contributions externally without properly maintaining
them,
> doesn't
> >>>>>>>>> make much sense to me.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> Max
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax
<
> mjsax@apache.org> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> many people are building quite exiting stuff
on top of Flink.
> It is hard
> >>>>>>>>>> to keep an good overview on what stuff is available
and what
> not. What
> >>>>>>>>>> do you think about starting a second git repository
> "flink-external"
> >>>>>>>>>> that collects all those code?
> >>>>>>>>>>
> >>>>>>>>>> The ideas would be to collect stuff in a central
point, such
> that people
> >>>>>>>>>> can access it easily and get an overview what
is already
> available (this
> >>>>>>>>>> might also avoid duplicate development). It
might also be a
> good point
> >>>>>>>>>> to show common patterns. In order to collect
as much as
> possible, the
> >>>>>>>>>> contributing requirement (with respect to testing
etc) could be
> lower
> >>>>>>>>>> than for Flink itself.
> >>>>>>>>>>
> >>>>>>>>>> For example, I recently started a small flink-clojure
module
> with a
> >>>>>>>>>> simple word-count example to answer a question
on SO. Including
> this in
> >>>>>>>>>> Flink would not be appropriate. However, for
a flink-external
> repro it
> >>>>>>>>>> might be nice to have.
> >>>>>>>>>>
> >>>>>>>>>> What do you think about it?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -Matthias
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Regards,
> >>>>>> Chiwan Park
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message