Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Message-ID: <5617C146.708@apache.org>
Date: Fri, 09 Oct 2015 15:29:42 +0200
From: "Matthias J. Sax" <mjsax@apache.org>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Icedove/31.8.0
MIME-Version: 1.0
To: dev@flink.apache.org
Subject: Re: [DISCUSS] flink-external
References: <5614E8FA.70208@apache.org>
 <CAGco--bpgz1XqrcHZdW-kjgn0JobS1ExqFCNfZY3mwJ10KhgrA@mail.gmail.com>
 <CAJZ2dcWL17boidAQxhMa8hSM9M2QWx9cf2YoSmaGLUDSKYD_tw@mail.gmail.com>
 <56164542.60209@apache.org> <2F276189-2C43-4A63-AFF0-EEF2B699F69B@apache.org>
 <CAGco--aHk1_vxzMNcUt8k2ERTsDn3Ex=m256RP=ti=DdKNOuow@mail.gmail.com>
 <561660D5.6040208@apache.org>
 <CAGco--ZHaVjy9PwEBjHF1f7COTaFutTouqiKJpEt0DdXD6ranQ@mail.gmail.com>
 <5617B31C.6040301@apache.org>
 <CAGco--bV6WmfTeunK+9kvbkoN3yMA2+oOwp7DSiXq8x8gr2aJQ@mail.gmail.com>
In-Reply-To: 
 <CAGco--bV6WmfTeunK+9kvbkoN3yMA2+oOwp7DSiXq8x8gr2aJQ@mail.gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
 protocol="application/pgp-signature";
 boundary="bmLOGF2o4vBamU4Knh81t4rj9HdCcEMjv"

--bmLOGF2o4vBamU4Knh81t4rj9HdCcEMjv
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

I was not sure what we should add and was hoping for input from the
community.

I am aware of the following projects we might want to add:

  - Zeppelin
  - SAMOA
  - Mahout
  - Cascading (dataartisan repo)
  - BigPetStore
  - Gradoop


-Matthias


On 10/09/2015 03:07 PM, Maximilian Michels wrote:
> Cool. Right now the list is empty. Do you already have a list you
> could include in the upcoming pull request? :)
>=20
> On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <mjsax@apache.org> wrot=
e:
>> Hi,
>>
>> I just started this. Please see
>> https://github.com/mjsax/flink-web/tree/flink-external-page
>>
>> I think, it is the best way to extend the "Downloads" page. I would al=
so
>> add a link to this on the main page's "Getting Started" section.
>>
>> As a first try, I started like this:
>>> Third party packages
>>>
>>> This is a list of third party packages (ie, libraries, system extensi=
ons, or examples) build for Flink. The Flink community only collects link=
s to those packages but does not maintain them. Thus, they do not belong =
to the Apache Flink project and the community cannot give any support for=
 them.
>>> Package Name
>>>
>>> Available for Flink 0.8.x and 0.9.x
>>>
>>> Short description
>>>
>>> Please let us know, if we missed to list your package. Be aware, that=
 we might remove listed packages without notice.
>>
>> Can you please give me some input, what projects I should add initiall=
y?
>>
>>
>> -Matthias
>>
>>
>> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
>>> IMHO we can do that. There should be a disclaimer that the third part=
y
>>> software is not officially supported.
>>>
>>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mjsax@apache.org> wr=
ote:
>>>> Should we add a new page at Flink project web page?
>>>>
>>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
>>>>> +1 for your pragmatic approach, Vasia. A simple collection of third=

>>>>> party software using Flink should be enough for now; of course,
>>>>> outside the Apache realm.
>>>>>
>>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanpark@apache.org=
> wrote:
>>>>>> +1 for Vasia=E2=80=99s suggestion. From a long-term perspective, t=
he site like Spark Packages [1] would be helpful to manage external contr=
ibution.
>>>>>>
>>>>>> [1] http://spark-packages.org
>>>>>>
>>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mjsax@apache.org> w=
rote:
>>>>>>>
>>>>>>> Thanks for the feedback.
>>>>>>>
>>>>>>> I think, the repository does not need to build on a single Flink
>>>>>>> release. From my point of view, there should be a single parent m=
odule
>>>>>>> that contains *independent modules* for each extension/library (t=
here
>>>>>>> should be no "cross dependencies" between the modules and each mo=
dule
>>>>>>> can specify the flink dependencies it needs by itself). This make=
 is
>>>>>>> most flexible. And if a library works on an old release, it might=
 just
>>>>>>> stay there as is. If a library changes (due to Flink changes), it=
 might
>>>>>>> just be contained multiple times for different Flink releases.
>>>>>>>
>>>>>>> Each module should provide a short doc (README) that shows how to=
 use an
>>>>>>> integrate it with Flink. Thus, the responsibility goes to the
>>>>>>> contributor to maintain the library. If it breaks and is not main=
tained
>>>>>>> any further, we can simple remove it.
>>>>>>>
>>>>>>> I agree, that the community might not be able to maintain those
>>>>>>> extension/libraries right now. I would put the responsibility (mo=
re or
>>>>>>> less completely) on the contributor and delete project that do no=
t fix
>>>>>>> any more.
>>>>>>>
>>>>>>> @Vasia: a link to a library could be included in the README. If a=
nybody
>>>>>>> only wants to share a library but not contribute code, the parent=
 README
>>>>>>> could contain a list of additional links.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>> On 10/08/2015 12:15 PM, Vasiliki Kalavri wrote:
>>>>>>>> How about, for now, we simply create a page where we gather link=
s/short
>>>>>>>> descriptions of all these contributions
>>>>>>>> and let the maintenance and dependency management to the tool/li=
brary
>>>>>>>> creators?
>>>>>>>> This way we will at least have these contributions in one place =
and link to
>>>>>>>> them somewhere from the website.
>>>>>>>>
>>>>>>>> -Vasia.
>>>>>>>>
>>>>>>>> On 8 October 2015 at 12:06, Maximilian Michels <mxm@apache.org> =
wrote:
>>>>>>>>
>>>>>>>>> Hi Matthias,
>>>>>>>>>
>>>>>>>>> Thanks for bringing up this idea. Actually, it has been discuss=
ed a
>>>>>>>>> couple of times on the mailing list whether we should have a ce=
ntral
>>>>>>>>> place for third-party extensions/contributions/libraries. This =
could
>>>>>>>>> either be something package-based or, like you proposed, anothe=
r
>>>>>>>>> repository.
>>>>>>>>>
>>>>>>>>> An external place for contributions raises a couple of question=
s
>>>>>>>>>
>>>>>>>>> - Which version should the external contributions be based on?
>>>>>>>>> - How do we make sure, the extensions are continuously updated?=

>>>>>>>>> (dedicated maintainers or automatic compatibility checks)
>>>>>>>>> - How do we easily plug-in the external modules into Flink?
>>>>>>>>>
>>>>>>>>> In the long term, we really need a solution for these questions=
=2E The
>>>>>>>>> code base of Flink is growing and more and more packages go to
>>>>>>>>> flink-contrib/flink-staging. I would find something packaged-ba=
sed
>>>>>>>>> better than a repository. Quite frankly, momentarily, I think
>>>>>>>>> developing such a plugin system is out of scope for most Flink
>>>>>>>>> developers. At the current pace of Flink development, collectin=
g these
>>>>>>>>> contributions externally without properly maintaining them, doe=
sn't
>>>>>>>>> make much sense to me.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Max
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax <mjsax@apache.=
org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> many people are building quite exiting stuff on top of Flink. =
It is hard
>>>>>>>>>> to keep an good overview on what stuff is available and what n=
ot. What
>>>>>>>>>> do you think about starting a second git repository "flink-ext=
ernal"
>>>>>>>>>> that collects all those code?
>>>>>>>>>>
>>>>>>>>>> The ideas would be to collect stuff in a central point, such t=
hat people
>>>>>>>>>> can access it easily and get an overview what is already avail=
able (this
>>>>>>>>>> might also avoid duplicate development). It might also be a go=
od point
>>>>>>>>>> to show common patterns. In order to collect as much as possib=
le, the
>>>>>>>>>> contributing requirement (with respect to testing etc) could b=
e lower
>>>>>>>>>> than for Flink itself.
>>>>>>>>>>
>>>>>>>>>> For example, I recently started a small flink-clojure module w=
ith a
>>>>>>>>>> simple word-count example to answer a question on SO. Includin=
g this in
>>>>>>>>>> Flink would not be appropriate. However, for a flink-external =
repro it
>>>>>>>>>> might be nice to have.
>>>>>>>>>>
>>>>>>>>>> What do you think about it?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Chiwan Park
>>>>>>
>>>>>>
>>>>>>
>>>>
>>


--bmLOGF2o4vBamU4Knh81t4rj9HdCcEMjv
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJWF8FGAAoJEFCVK48prEZ4P1kP/2D+Cpi4zQm9Fmzr2t7hXaGz
nOnfirl1hhTWA09hdoGEm/bJ1oV5LEIJdpR5VqTqZhCSTaznzpbghgEu7c9RdGq6
ieSjjWzCSj672x6rSuamfILK5m9xUF6XGWyM5L1CqUqIQsDVqsxS1f4wNmpS7CqD
hpbvgcUEYYu0ATTglIEnQDKvdg5oQ3k6b1iGfmUY16Kf8ysyHEuzIUHyDGw7/R4Y
OL3kDwzuhZsyxhAvbmvXH7gjJRcHyyM7bi/fj/QSRlvd1QRjZ1i/qmA9nENsqGpP
MLuCquudFcAjfeO7pE6MSBKoYN/WnwEpwuY+j5VbiSZQPH0clxybJD9bCOncrVbu
m0+r2Fr368eIlurI0FEIWk2CUmOWxm6MvblD35yRlO6ka70YAXhtB2NcyHccvP3g
iQOJHwE8IhZV2/0WYW6eCCkmZK2Az0Z8cN/aVXv8Y77ApCVMkrp163G34d4sVHC5
eluL7Gc9xRtburERyjPBArhnpga0E7GeSyG2d/0/EICU5t3lNqp86mWF2HS3tnQE
BbUPadT+Kv1ooRs+Zd94uY3ubs94tI5oVcQDjBHh1lsJ7JUs6XyT7+x51Mxa83vO
JtUCcphJdoX2avL09hY2cOPNfhngK+GdbGsyMXH201hlcqVJVT5SfugWZUDzZx4b
AtyCkC9DcXy9uhDHD2gz
=PFI+
-----END PGP SIGNATURE-----

--bmLOGF2o4vBamU4Knh81t4rj9HdCcEMjv--