Mailing-List: contact user-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@beam.apache.org
MIME-Version: 1.0
In-Reply-To: <CAP76Zfg2nh1YtWr-JuetSL27M4nH2XchDxXoq-W_oiKD6HruMQ@mail.gmail.com>
References: <CAP76Zfh1OrNi-2=SwDSpRVuJ-KZC+vLYPHYD8=G6iL5jxbZ-zQ@mail.gmail.com>
 <CAF9t7_7jBdm6udAumgZnEX81Wcvneoqq8Tp5FUWwuNMGBdoZQw@mail.gmail.com> <CAP76Zfg2nh1YtWr-JuetSL27M4nH2XchDxXoq-W_oiKD6HruMQ@mail.gmail.com>
From: Lukasz Cwik <lcwik@google.com>
Date: Thu, 5 Oct 2017 10:58:53 -0700
Message-ID: <CAF9t7_4qiatTxEyxX29pPSH6v0PmQMEKjtc1YV=q55Dg6dWrWg@mail.gmail.com>
Subject: Re: Best way to inject external per key config into the pipeline
To: user@beam.apache.org
Content-Type: multipart/alternative; boundary="001a1145311c38805a055ad07a2e"
archived-at: Thu, 05 Oct 2017 17:58:59 -0000

--001a1145311c38805a055ad07a2e
Content-Type: text/plain; charset="UTF-8"

Yes, everytime you start the pipeline you need to read in all the config
data. You can do this by flattening a bounded source which reads the
current state of config data with a streaming source that gets updates and
then use that as a side input. On the other hand, with a few million keys,
using an in memory cache with your own refresh policy that contacts your
datastore would also likely work well.

On Thu, Oct 5, 2017 at 10:16 AM, Yihua Fang <fang.yihua.eric@gmail.com>
wrote:

> I thought about streaming the updates and using side input, but since the
> config data are not persisted in the data pipeline, how would the config
> being populated in the first place. Is the solution to populate through
> streaming every time the pipeline is rebooted?
>
> 1. The config data can be a few minutes to a few hours late. No need to
> immediately reflects the config.
> 2. The config data shouldn't be change often. It is configured by human
> users.
> 3. The config data per key should be about 10-20 key value pairs.
> 4. Ideally the key number is in the range of a few millions, but a few
> thousands to begin with.
>
> Thanks
> Eric
>
> On Thu, Oct 5, 2017 at 9:09 AM Lukasz Cwik <lcwik@google.com> wrote:
>
>> Can you stream the updates to the keys into the pipeline and then use it
>> as a side input performing a join against on your main stream that needs
>> the config data?
>> You could also use an in memory cache that periodically refreshes keys
>> from the external source.
>>
>> A better answer depends on:
>> * how stale can the config data be?
>> * how often does the config data change?
>> * how much config data you expect to have per key?
>> * how many keys do you expect to have?
>>
>>
>> On Wed, Oct 4, 2017 at 5:41 PM, Yihua Fang <fang.yihua.eric@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> The use case is that I have an external source that store a
>>> configuration for each key accessible via restful APIs and Beam pipeline
>>> should use the config to process each element for each key. What is the
>>> best approach to facilitate injecting the latest config into the pipeline?
>>>
>>> Thanks
>>> Eric
>>>
>>
>>

--001a1145311c38805a055ad07a2e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Yes, everytime you start the pipeline you need to read in =
all the config data. You can do this by flattening a bounded source which r=
eads the current state of config data with a streaming source that gets upd=
ates and then use that as a side input. On the other hand, with a few milli=
on keys, using an in memory cache with your own refresh policy that contact=
s your datastore would also likely work well.</div><div class=3D"gmail_extr=
a"><br><div class=3D"gmail_quote">On Thu, Oct 5, 2017 at 10:16 AM, Yihua Fa=
ng <span dir=3D"ltr">&lt;<a href=3D"mailto:fang.yihua.eric@gmail.com" targe=
t=3D"_blank">fang.yihua.eric@gmail.com</a>&gt;</span> wrote:<br><blockquote=
 class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli=
d;padding-left:1ex"><div dir=3D"ltr">I thought about streaming the updates =
and using side input, but since the config data are not persisted in the da=
ta pipeline, how would the config being populated in the first place. Is th=
e solution to populate through streaming every time the pipeline is reboote=
d?<div><br></div><div>1. The config data can be a few minutes to a few hour=
s late. No need to immediately reflects the config.</div><div>2. The config=
 data shouldn&#39;t be change often. It is configured by human users.</div>=
<div>3. The config data per key should be about 10-20 key value pairs.</div=
><div>4. Ideally the key number is in the range of a few millions, but a fe=
w thousands to begin with.</div><div><br></div><div>Thanks</div><div>Eric</=
div></div><div class=3D"HOEnZb"><div class=3D"h5"><br><div class=3D"gmail_q=
uote"><div dir=3D"ltr">On Thu, Oct 5, 2017 at 9:09 AM Lukasz Cwik &lt;<a hr=
ef=3D"mailto:lcwik@google.com" target=3D"_blank">lcwik@google.com</a>&gt; w=
rote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex=
;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Can you stre=
am the updates to the keys into the pipeline and then use it as a side inpu=
t performing a join against on your main stream that needs the config data?=
<div>You could also use an in memory cache that periodically refreshes keys=
 from the external source.<br><div><br></div><div>A better answer depends o=
n:</div><div>* how stale can the config data be?</div><div>* how often does=
 the config data change?</div><div>* how much config data you expect to hav=
e per key?</div><div>* how many keys do you expect to have?</div><div><br><=
/div></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">=
On Wed, Oct 4, 2017 at 5:41 PM, Yihua Fang <span dir=3D"ltr">&lt;<a href=3D=
"mailto:fang.yihua.eric@gmail.com" target=3D"_blank">fang.yihua.eric@gmail.=
com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr=
">Hi,<div><br></div><div>The use case is that I have an external source tha=
t store a configuration for each key accessible via restful APIs and Beam p=
ipeline should use the config to process each element for each key. What is=
 the best approach to facilitate injecting the latest config into the pipel=
ine?</div><div><br></div><div>Thanks</div><div>Eric</div></div>
</blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>

--001a1145311c38805a055ad07a2e--