flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Future of Queryable State Feature
Date Mon, 20 Mar 2017 14:25:04 GMT
I think adding this could be as simple as adding a makeWindowContentsQueryable(String) call
to WindowedStream. In there, we create the StateDescriptor for keeping the window contents
so we can call setQueryable() on the StateDescriptor and set the name under which it should
be accessible. If you want you can create an issue and change this. I can help you if you
have any questions.

For custom Triggers, I think you can also call setQueryable() on a StateDescriptor.

The tricky thing when it comes to windows is that state is scoped to a window, so the querying
logic has to take that into account.

I also cc’ed Ufuk and Nico, who probably know best what’s going on with queryable state.
> On 16 Mar 2017, at 15:33, Joe Olson <jo4243@outlook.com> wrote:
> 
> I have a question regarding the future direction of the queryable state feature.
> 
> 
> We are current using this feature in production implemented in a RichFlatMap. It is doing
what we need it to do at the scale we need it done in, with the resources we have assigned
to it. Win.
> 
> 
> However, we'd also like to use this feature in conjunction with Flink's windowing. The
"Rich" execution environment is not exposed in any of the windows / triggers / apply hierarchy,
so we cannot expose any of the state managed within the windows outside of Flink. Many of
our use cases require us to have access to values as they are being accumulated, as well as
the aggregated result.
> 
> 
> We can get by with the RichFlatMap for now. I'd like some clarification as to whether
or not the queryable state feature is going to be extended to the windowing components for
the next milestone release. This will determine our Flink development milestones for the next
few months. From consulting the open items in JIRA, it does not look like it is on the docket.
> 
> 
> I'd be more than willing to help out implementing this feature, but I don't think I have
the experience to submit this change on my own.


Mime
View raw message