Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <CAAdrtT3PSZXU-=jh+p4P+55MUEQnPO5vBe0LjEroy3E+X1N=3Q@mail.gmail.com>
References: <CAELUF_D_am5qCxwAteCL6sEGX+L5E6R3B62SUgAgkkw_PV+O_g@mail.gmail.com>
 <CAAdrtT3PSZXU-=jh+p4P+55MUEQnPO5vBe0LjEroy3E+X1N=3Q@mail.gmail.com>
From: Flavio Pompermaier <pompermaier@okkam.it>
Date: Thu, 5 May 2016 10:08:11 +0200
Message-ID: <CAELUF_BezMME4Zc6R2b7Yzm-EzvMY20hQsCNiB1H6PVvToLJBQ@mail.gmail.com>
Subject: Re: Discussion about a Flink DataSource repository
To: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary=047d7b66f41b0cd3a4053213d9e3
archived-at: Thu, 05 May 2016 08:08:41 -0000

--047d7b66f41b0cd3a4053213d9e3
Content-Type: text/plain; charset=UTF-8

HI Fabian,
thanks for your detailed answer, as usual ;)

I think that an external service it's ok,actually I wasn't aware of the
TableSource interface.
As you said, an utility to serialize and deserialize them would be very
helpful and will ease this thing.
However, registering metadata for a table is a very common task to do.
Wouldn't be of useful for other Flink-related projects (I was thinking to
Nifi for example) to define a common minimal set of (optional) metadata to
display in a UI for a TableSource (like name, description, creationDate,
creator, field aliases)?

About point 2, I think that dataset broadcasting or closure variables are
useful when you write a program, not if you try to "compose" it using
reusable UDFs (using a script like in Pig).
Of course, the worst case scenario for us (e.g. right now) is to connect to
our repository within rich operators but I thought that it could be easy to
define a link from operators to TableEnvironment and then to TableSource
(using the lineage tag/source-id you said) and, finally to its metadata. I
don't know whether this is specific only to us, I just wanted to share our
needs and see if the table API development could benefit from them.

Best,
Flavio

On Wed, May 4, 2016 at 10:35 AM, Fabian Hueske <fhueske@gmail.com> wrote:

> Hi Flavio,
>
> I thought a bit about your proposal. I am not sure if it is actually
> necessary to integrate a central source repository into Flink. It should be
> possible to offer this as an external service which is based on the
> recently added TableSource interface. TableSources could be extended to be
> able to serialize and descerialize their configuration to/from JSON. When
> the external repository service starts, it can read the JSON fields and
> instantiate and register TableSource objectes. The repository could also
> hold metadata about the sources and serve a (web) UI to list available
> source. When a Flink program wants to access a data source which is
> registered in the repository, it could lookup the respective TableSouce
> object from the repository.
>
> Given that an integration of metadata with Flink user functions (point 2.
> in your proposal) is a very special requirement, I am not sure how much
> "native" support should be added to Flink. Would it be possible to add a
> lineage tag to each record and ship the metadata of all sources as
> broadcast set to each operator? Then user functions could lookup the
> metadata from the broadcast set.
>
> Best, Fabian
>
> 2016-04-29 12:49 GMT+02:00 Flavio Pompermaier <pompermaier@okkam.it>:
>
>> Hi to all,
>>
>> as discussed briefly with Fabian, for our products in Okkam we need a
>> central repository of DataSources processed by Flink.
>> With respect to existing external catalogs, such as Hive or Confluent's
>> SchemaRegistry, whose objective is to provide necessary metadata to
>> read/write the registered tables, we would also need a way to acess to
>> other general metadata (e.g. name, description, creator, creation date,
>> lastUpdate date, processedRecords, certificationLevel of provided data,
>> provenance, language, etc).
>>
>> This integration has 2 main goals:
>>
>>    1. In a UI: to enable the user to choose (or even create) a
>>    datasource to process with some task (e.g. quality assessment) and then see
>>    its metadata (name, description,  creator user, etc)
>>    2. During a Flink job: when 2 datasource gets joined and we have
>>    multiple values for an attribute (e.g. name or lastname) we can access the
>>    datasource metadata to decide which value to retain (e.g. the one coming
>>    from the most authoritative/certified source for that attribute)
>>
>> We also think that this could be of interest for projects like Apache
>> Zeppelin or Nifi enabling them to suggest to the user the sources to start
>> from.
>>
>> Do you think it makes sense to think about designing such a module for
>> Flink?
>>
>> Best,
>> Flavio
>>
>
>

--047d7b66f41b0cd3a4053213d9e3
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div>HI Fabian, <br></div>thanks for your d=
etailed answer, as usual ;)<br><br></div>I think that an external service i=
t&#39;s ok,actually I wasn&#39;t aware of the TableSource interface.<br></d=
iv>As you said, an utility to serialize and deserialize them would be very =
helpful and will ease this thing.<br></div><div>However, registering metada=
ta for a table is a very common task to do. Wouldn&#39;t be of useful for o=
ther Flink-related projects (I was thinking to Nifi for example) to define =
a common minimal set of (optional) metadata to display in a UI for a TableS=
ource (like name, description, creationDate, creator, field aliases)?<br></=
div><div><br>About point 2, I think that dataset broadcasting or closure va=
riables are useful when you write a program, not if you try to &quot;compos=
e&quot; it using reusable UDFs (using a script like in Pig). <br>Of course,=
 the worst case scenario for us (e.g. right now) is to connect to our repos=
itory within rich operators but I thought that it could be easy to define a=
 link from operators to TableEnvironment and then to TableSource (using the=
 lineage tag/source-id you said) and, finally to its metadata. I don&#39;t =
know whether this is specific only to us, I just wanted to share our needs =
and see if the table API development could benefit from them.<br><br></div>=
<div>Best,<br></div><div>Flavio<br></div><div class=3D"gmail_extra"><br><di=
v class=3D"gmail_quote">On Wed, May 4, 2016 at 10:35 AM, Fabian Hueske <spa=
n dir=3D"ltr">&lt;<a href=3D"mailto:fhueske@gmail.com" target=3D"_blank">fh=
ueske@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" =
style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><di=
v dir=3D"ltr"><div><div>Hi Flavio,<br><br></div><div>I thought a bit about =
your proposal. I am not sure if it is actually necessary to integrate a cen=
tral source repository into Flink. It should be possible to offer this as a=
n external service which is based on the recently added TableSource interfa=
ce. TableSources could be extended to be able to serialize and descerialize=
 their configuration to/from JSON. When the external repository service sta=
rts, it can read the JSON fields and instantiate and register TableSource o=
bjectes. The repository could also hold metadata about the sources and serv=
e a (web) UI to list available source. When a Flink program wants to access=
 a data source which is registered in the repository, it could lookup the r=
espective TableSouce object from the repository.<br><br></div><div>Given th=
at an integration of metadata with Flink user functions (point 2. in your p=
roposal) is a very special requirement, I am not sure how much &quot;native=
&quot; support should be added to Flink. Would it be possible to add a line=
age tag to each record and ship the metadata of all sources as broadcast se=
t to each operator? Then user functions could lookup the metadata from the =
broadcast set.<br><br></div><div>Best, Fabian<br></div></div></div><div cla=
ss=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br><div class=
=3D"gmail_quote">2016-04-29 12:49 GMT+02:00 Flavio Pompermaier <span dir=3D=
"ltr">&lt;<a href=3D"mailto:pompermaier@okkam.it" target=3D"_blank">pomperm=
aier@okkam.it</a>&gt;</span>:<br><blockquote class=3D"gmail_quote" style=3D=
"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D=
"ltr">Hi to all,<div><div dir=3D"ltr"><p></p><p></p><p></p><p></p></div></d=
iv>
<div>as discussed briefly with Fabian, for our products in Okkam we need a =
central repository of DataSources processed by Flink.</div><div>With respec=
t to existing external catalogs, such as Hive or=C2=A0<span style=3D"font-s=
ize:12.8px">Confluent&#39;s SchemaRegistry, whose objective is to provide n=
ecessary metadata to read/write the registered tables, we would also need a=
 way to acess to other general metadata (e.g. name, description, creator, c=
reation date, lastUpdate date, processedRecords, certificationLevel of prov=
ided data, provenance, language, etc).</span></div><div><span style=3D"font=
-size:12.8px"><br></span></div><div><span style=3D"font-size:12.8px">This i=
ntegration has 2 main goals:</span></div><div><ol><li><span style=3D"font-s=
ize:12.8px">In a UI: to enable the user to choose (or even create) a dataso=
urce to process with some task (e.g. quality assessment) and then see its m=
etadata (name, description, =C2=A0creator user, etc)</span></li><li><span s=
tyle=3D"font-size:12.8px">During a Flink job: when 2 datasource gets joined=
 and we have multiple values for an attribute (e.g. name or lastname) we ca=
n access the datasource metadata to decide which value to retain (e.g. the =
one coming from the most authoritative/certified source for that attribute)=
</span></li></ol><div><span style=3D"font-size:12.8px">We also think that t=
his could be of interest for projects like Apache Zeppelin or Nifi enabling=
 them to suggest to the user the sources to start from.</span></div></div><=
div><span style=3D"font-size:12.8px"><br></span></div><div><span style=3D"f=
ont-size:12.8px">Do you think it makes sense to think about designing such =
a module for Flink?</span></div><div><span style=3D"font-size:12.8px"><br><=
/span></div><div><span style=3D"font-size:12.8px">Best,</span></div><div><s=
pan style=3D"font-size:12.8px">Flavio</span></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br>
</div></div>

--047d7b66f41b0cd3a4053213d9e3--