Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Subject: Re: Future of Queryable State Feature
From: Aljoscha Krettek <aljoscha@apache.org>
In-Reply-To: <MWHPR13MB1565B4D011FD5454007FBC83D8260@MWHPR13MB1565.namprd13.prod.outlook.com>
Date: Mon, 20 Mar 2017 15:25:04 +0100
Cc: Ufuk Celebi <uce@apache.org>,
 nico@data-artisans.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <362C780C-9672-4DBD-B3F1-4EE7D1DB4CA6@apache.org>
References: <MWHPR13MB1565B4D011FD5454007FBC83D8260@MWHPR13MB1565.namprd13.prod.outlook.com>
To: dev@flink.apache.org
archived-at: Mon, 20 Mar 2017 14:25:10 -0000

I think adding this could be as simple as adding a =
makeWindowContentsQueryable(String) call to WindowedStream. In there, we =
create the StateDescriptor for keeping the window contents so we can =
call setQueryable() on the StateDescriptor and set the name under which =
it should be accessible. If you want you can create an issue and change =
this. I can help you if you have any questions.

For custom Triggers, I think you can also call setQueryable() on a =
StateDescriptor.

The tricky thing when it comes to windows is that state is scoped to a =
window, so the querying logic has to take that into account.

I also cc=E2=80=99ed Ufuk and Nico, who probably know best what=E2=80=99s =
going on with queryable state.
> On 16 Mar 2017, at 15:33, Joe Olson <jo4243@outlook.com> wrote:
>=20
> I have a question regarding the future direction of the queryable =
state feature.
>=20
>=20
> We are current using this feature in production implemented in a =
RichFlatMap. It is doing what we need it to do at the scale we need it =
done in, with the resources we have assigned to it. Win.
>=20
>=20
> However, we'd also like to use this feature in conjunction with =
Flink's windowing. The "Rich" execution environment is not exposed in =
any of the windows / triggers / apply hierarchy, so we cannot expose any =
of the state managed within the windows outside of Flink. Many of our =
use cases require us to have access to values as they are being =
accumulated, as well as the aggregated result.
>=20
>=20
> We can get by with the RichFlatMap for now. I'd like some =
clarification as to whether or not the queryable state feature is going =
to be extended to the windowing components for the next milestone =
release. This will determine our Flink development milestones for the =
next few months. =46rom consulting the open items in JIRA, it does not =
look like it is on the docket.
>=20
>=20
> I'd be more than willing to help out implementing this feature, but I =
don't think I have the experience to submit this change on my own.