flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Meghashyam Sandeep V <vr1meghash...@gmail.com>
Subject Re: static/dynamic lookups in flink streaming
Date Wed, 21 Dec 2016 17:46:50 GMT
As a follow up question, can we populate the operator state from an
external source?

My use case is as follows: I have a flink streaming process with Kafka as a
source. I only have ids coming from kafka messages. My look ups (<id,name>)
which is a static map come from a different source. I would like to use
those lookups while applying operators on stream from Kafka.

Thanks,
Sandeep

On Wed, Dec 21, 2016 at 6:17 AM, Fabian Hueske <fhueske@gmail.com> wrote:

> OK, I see. Yes, you can do that with Flink. It's actually a very common
> use case.
>
> You can store the names in operator state and Flink takes care of
> checkpointing the state and restoring it in case of a failure.
> In fact, the operator state is persisted in the state backends you
> mentioned before.
>
> Best, Fabian
>
> 2016-12-21 15:02 GMT+01:00 Meghashyam Sandeep V <vr1meghashyam@gmail.com>:
>
>> Hi Fabian,
>>
>> I meant look ups like IDs to names. For example if I have IDs coming
>> through the stream and if I want to replace them with corresponding names
>> stored in cache or somewhere within flink.
>>
>> Thanks,
>> Sandeep
>>
>> On Dec 21, 2016 12:35 AM, "Fabian Hueske" <fhueske@gmail.com> wrote:
>>
>>> Hi Sandeep,
>>>
>>> I'm sorry but I think I do not understand your question.
>>> What do you mean by static or dynamic look ups? Do you want to access an
>>> external data store and cache data?
>>>
>>> Can you give a bit more detail about your use?
>>>
>>> Best, Fabian
>>>
>>> 2016-12-20 23:07 GMT+01:00 Meghashyam Sandeep V <vr1meghashyam@gmail.com
>>> >:
>>>
>>>> Hi there,
>>>>
>>>> I know that there are various state backends to persist state. Is there
>>>> a similar way to persist static/dynamic look ups and use them while
>>>> streaming the data in Flink?
>>>>
>>>> Thanks,
>>>> Sandeep
>>>>
>>>>
>>>
>

Mime
View raw message