flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oytun Tez <oy...@motaword.com>
Subject Re: Making broadcast state queryable?
Date Fri, 06 Sep 2019 14:14:42 GMT
Hi Yu,

Excuse my late reply... We simply want Flink to be our centralized stream
analysis platform, where we 1) consume data, 2) generate analysis, 3)
present analysis. I honestly don't want "stream analysis" to spill out to
other components in our ecosystem (e.g., sinking insights into a DB-like
place).

So the case for QS for us is centralization of input, output, presentation.
State Processor API for instance also counts as a presentation tool for us
(on top of migration tool).

This kind of all-in-one (in, out, ui) packaging helps with small teams.

---
Oytun Tez

*M O T A W O R D*
The World's Fastest Human Translation Platform.
oytun@motaword.com — www.motaword.com


On Wed, Aug 14, 2019 at 3:13 AM Yu Li <carp84@gmail.com> wrote:

> Good to know your thoughts and the coming talk in Flink Forward Berlin
> Oytun, interesting topic and great job! And it's great to hear the voice
> from application perspective.
>
> Could you share, if possible, the reason why you need to open the state to
> outside instead of writing the result to sink and directly query there? In
> another thread there's a case that sink writes to different kafka topics so
> state is the only place to get a global view, is this the same case you're
> facing? Or some different requirements? I believe more attention will be
> drawn to QS if more and more user requirements emerge (smile).
>
> Thanks.
>
> Best Regards,
> Yu
>
>
> On Wed, 14 Aug 2019 at 00:50, Oytun Tez <oytun@motaword.com> wrote:
>
>> Thank you for the honest response, Yu!
>>
>> There is so much that comes to mind when we look at Flink as a
>> "application framework" (my talk
>> <https://europe-2019.flink-forward.org/conference-program#not-so-big-%E2%80%93-flink-as-a-true-application-framework>
>> in Flink Forward in Berlin will be about this). QS is one of them
>> (need-wise, not QS itself necessarily). I opened the design doc Vino Yang
>> created, I'll try to contribute to that.
>>
>> Meanwhile, for the sake of opening our state to outside, we will put a
>> stupid simple operator in between to keep a *duplicate* of the state...
>>
>> Thanks again!
>>
>>
>>
>>
>>
>> ---
>> Oytun Tez
>>
>> *M O T A W O R D*
>> The World's Fastest Human Translation Platform.
>> oytun@motaword.com — www.motaword.com
>>
>>
>> On Tue, Aug 13, 2019 at 6:29 PM Yu Li <carp84@gmail.com> wrote:
>>
>>> Hi Oytun,
>>>
>>> Sorry but TBH such support will probably not be added in the foreseeable
>>> future due to lack of committer bandwidth (not only support queryable
>>> broadcast state but all about QueryableState module) as pointed out in
>>> other threads [1] [2].
>>>
>>> However, I think you could open a JIRA for this so when things changed
>>> it could be taken care of. Thanks.
>>>
>>> [1] https://s.apache.org/MaOl
>>> [2] https://s.apache.org/r8k8a
>>>
>>> Best Regards,
>>> Yu
>>>
>>>
>>> On Tue, 13 Aug 2019 at 20:34, Oytun Tez <oytun@motaword.com> wrote:
>>>
>>>> Hi there,
>>>>
>>>> Can we set a broadcast state as queryable? I've looked around, not much
>>>> to find about it. I am receiving UnknownKvStateLocation when I try to query
>>>> with the descriptor/state name I give to the broadcast state.
>>>>
>>>> If it doesn't work, what could be the alternative? My mind goes around
>>>> ctx.getBroadcastState and making it queryable in the operator level (I'd
>>>> rather not).
>>>>
>>>> ---
>>>> Oytun Tez
>>>>
>>>> *M O T A W O R D*
>>>> The World's Fastest Human Translation Platform.
>>>> oytun@motaword.com — www.motaword.com
>>>>
>>>

Mime
View raw message