Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <etPan.591e85f0.6487549d.371a@apache.org>
References: <CADrk5qNzxn=SQgL1GQbojOY8aR26RLm60A43uEJZhAemxyZwMw@mail.gmail.com>
 <etPan.591e85f0.6487549d.371a@apache.org>
From: Fabian Hueske <fhueske@gmail.com>
Date: Fri, 19 May 2017 11:08:57 +0200
Message-ID: <CAAdrtT3uEo4f1x51j8otaaTPqmyzuR6NBWekM49OAQB2WkOFQw@mail.gmail.com>
Subject: Re: Best practices to maintain reference data for Flink Jobs
To: "Tzu-Li (Gordon) Tai" <tzulitai@apache.org>
Cc: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="001a114e4b38dedd32054fdce0c2"
archived-at: Fri, 19 May 2017 09:09:33 -0000

--001a114e4b38dedd32054fdce0c2
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

+1 to what Gordon said.

Queryable state is rather meant as an external interface to streaming jobs
than for lookups within jobs.
Accessing co-located state should give you better performance and is
probably easier to implement and maintain.

Cheers,
Fabian

2017-05-19 7:43 GMT+02:00 Tzu-Li (Gordon) Tai <tzulitai@apache.org>:

> Hi,
>
> Can the enriching data be keyed? Or is it something that has to be
> broadcasted to each operator?
> Either way, I think Side Inputs (an upcoming feature in the future) is th=
e
> best fit for this. You can take a look at https://issues.apache.org/
> jira/browse/FLINK-6131.
>
> Regarding the 3 options you listed:
>
> By using QueryableState in option B, what you mean is that you want to
> feed the enriching data stream to a separate job, let that job allow
> queryable state, and query that state from the actual application job
> operators, correct? If so, I think options A and B would mean the same
> thing; i.e., they require accessing data external to the job.
>
> If the enriching data can somehow be keyed with the stream that requires
> it, I would go for option C using connected streams, with the enriching
> data as one input and the actual data as the other. Instead of just
> =E2=80=9Ccaching the enriching data in memory=E2=80=9D, you should regist=
er it as a managed
> Link state for the CoMapFunction / CoFlatMapFunction. The actual input
> stream records can just access that registered state locally.
>
> Cheers,
> Gordon
>
>
> On 19 May 2017 at 7:11:07 AM, Sand Stone (sand.m.stone@gmail.com) wrote:
>
> Hi. Say I have a few reference data sets need to be used for a
> streaming job. The sizes range between 10M-10GB. The data is not
> static, will be refreshed at minutes and/or day intervals.
>
> With the new advancements in Flink, it seems there are quite a few
> options.
> A. Store all the data in an external (kv) database cluster. And use
> async io calls
> * data refresh can be done in a few different ways
> B. Use the new Querytable State feature
> * it seems there is no "easy" API to discover the
> queryable state at the moment. Need to use the restful API to figure
> out the job id.
> C. Ingest the reference data into the job and cache them in memory
> Any other option?
>
> On paper, it seems option B with the Queryable State is the cleanest
> solution.
>
> Any comment/suggestion is greatly appreciated in particular in terms
> of robustness and consistent recovery.
>
> Thanks much!
>
>

--001a114e4b38dedd32054fdce0c2
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div>+1 to what Gordon said.<br><br></div>Q=
ueryable state is rather meant as an external interface to streaming jobs t=
han for lookups within jobs.<br></div>Accessing co-located state should giv=
e you better performance and is probably easier to implement and maintain.<=
br><br></div>Cheers,<br></div>Fabian<br></div><div class=3D"gmail_extra"><b=
r><div class=3D"gmail_quote">2017-05-19 7:43 GMT+02:00 Tzu-Li (Gordon) Tai =
<span dir=3D"ltr">&lt;<a href=3D"mailto:tzulitai@apache.org" target=3D"_bla=
nk">tzulitai@apache.org</a>&gt;</span>:<br><blockquote class=3D"gmail_quote=
" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><=
div style=3D"word-wrap:break-word"><div id=3D"m_4604217439374648742bloop_cu=
stomfont" style=3D"font-family:Helvetica,Arial;font-size:13px;color:rgba(0,=
0,0,1.0);margin:0px;line-height:auto">Hi,</div><div id=3D"m_460421743937464=
8742bloop_customfont" style=3D"font-family:Helvetica,Arial;font-size:13px;c=
olor:rgba(0,0,0,1.0);margin:0px;line-height:auto"><br></div><div id=3D"m_46=
04217439374648742bloop_customfont" style=3D"font-family:Helvetica,Arial;fon=
t-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">Can the enri=
ching data be keyed? Or is it something that has to be broadcasted to each =
operator?</div><div id=3D"m_4604217439374648742bloop_customfont" style=3D"f=
ont-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;=
line-height:auto">Either way, I think Side Inputs (an upcoming feature in t=
he future) is the best fit for this. You can take a look at=C2=A0<a href=3D=
"https://issues.apache.org/jira/browse/FLINK-6131" target=3D"_blank">https:=
//issues.apache.org/<wbr>jira/browse/FLINK-6131</a>.</div><div id=3D"m_4604=
217439374648742bloop_customfont" style=3D"font-family:Helvetica,Arial;font-=
size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto"><br></div><div=
 id=3D"m_4604217439374648742bloop_customfont" style=3D"font-family:Helvetic=
a,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">R=
egarding the 3 options you listed:</div><div id=3D"m_4604217439374648742blo=
op_customfont" style=3D"font-family:Helvetica,Arial;font-size:13px;color:rg=
ba(0,0,0,1.0);margin:0px;line-height:auto"><br></div><div id=3D"m_460421743=
9374648742bloop_customfont" style=3D"font-family:Helvetica,Arial;font-size:=
13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">By using QueryableS=
tate in option B, what you mean is that you want to feed the enriching data=
 stream to a separate job, let that job allow queryable state, and query th=
at state from the actual application job operators, correct? If so, I think=
 options A and B would mean the same thing; i.e., they require accessing da=
ta external to the job.=C2=A0</div><div id=3D"m_4604217439374648742bloop_cu=
stomfont" style=3D"font-family:Helvetica,Arial;font-size:13px;color:rgba(0,=
0,0,1.0);margin:0px;line-height:auto"><br></div><div id=3D"m_46042174393746=
48742bloop_customfont" style=3D"font-family:Helvetica,Arial;font-size:13px;=
color:rgba(0,0,0,1.0);margin:0px;line-height:auto">If the enriching data ca=
n somehow be keyed with the stream that requires it, I would go for option =
C using connected streams, with the enriching data as one input and the act=
ual data as the other. Instead of just =E2=80=9Ccaching the enriching data =
in memory=E2=80=9D, you should register it as a managed Link state for the =
CoMapFunction / CoFlatMapFunction. The actual input stream records can just=
 access that registered state locally.</div> <div><br></div>Cheers,<div>Gor=
don<div><div class=3D"h5"><br> <div class=3D"m_4604217439374648742bloop_sig=
n" id=3D"m_4604217439374648742bloop_sign_1495171999318807040"></div> <br><p=
 class=3D"m_4604217439374648742airmail_on">On 19 May 2017 at 7:11:07 AM, Sa=
nd Stone (<a href=3D"mailto:sand.m.stone@gmail.com" target=3D"_blank">sand.=
m.stone@gmail.com</a>) wrote:</p> <blockquote type=3D"cite" class=3D"m_4604=
217439374648742clean_bq"><span><div><div></div><div>Hi. Say I have a few re=
ference data sets need to be used for a
<br>streaming job. The sizes range between 10M-10GB. The data is not
<br>static, will be refreshed at minutes and/or day intervals.
<br>
<br>With the new advancements in Flink, it seems there are quite a few opti=
ons.
<br>   A. Store all the data in an external (kv) database cluster. And use
<br>async io calls
<br>          * data refresh can be done in a few different ways
<br>   B. Use the new Querytable State feature
<br>            * it seems there is no &quot;easy&quot; API to discover the
<br>queryable state at the moment. Need to use the restful API to figure
<br>out the job id.
<br>   C. Ingest the reference data into the job and cache them in memory
<br>Any other option?
<br>
<br>On paper, it seems option B with the Queryable State is the cleanest so=
lution.
<br>
<br>Any comment/suggestion is greatly appreciated in particular in terms
<br>of robustness and consistent recovery.
<br>
<br>Thanks much!
<br></div></div></span></blockquote></div></div></div></div></blockquote></=
div><br></div>

--001a114e4b38dedd32054fdce0c2--