flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Kloudas <k.klou...@data-artisans.com>
Subject Re: Maintain heavy hitters in Flink application
Date Thu, 07 Dec 2017 10:10:24 GMT
Hi Max,

You are right that Queryable State is not designed to be used as a means for a job to query
its own state.
In fact, given that you do not know the jobId of your job from within the job itself, I do
not think you can use 
queryable state in your scenario.

What you can do is to have a flatMap computing the hot keys or heavy hitters, and emit as
main output the 
elements themselves for further processing, and as a side output the computed statistics.
The side output 
is a data stream itself so you can store it in an external storage system (e.g. a KV store)
and use AsyncIO to 
query that system downstream. This will solve the problem of having access to the state from
all tasks. 

This is a simple solution but I am not sure about the performance implications. 
You can try it to see if it actually fits your needs.

Thanks, 
Kostas


> On Dec 5, 2017, at 10:32 AM, Fabian Hueske <fhueske@gmail.com> wrote:
> 
> Hi,
> 
> I haven't done that before either. The query API will change with the next version (Flink
1.4.0) which is currently being prepared for releasing.
> Kostas (in CC) might be able to help you.
> 
> Best, Fabian
> 
> 2017-12-05 9:52 GMT+01:00 m@xi <makisntpap@gmail.com <mailto:makisntpap@gmail.com>>:
> Hi Fabian,
> 
> Thanks for your answer. Initially, I have excluded Queryable State as an
> option as it explicitly mentioned that it is used for querying state outside
> flink.
> 
> Now that I am reading the documentation I am not sure how I may achieve
> that. I have to set ports and addresses which I am not sure I should since I
> am reading the queryable state from inside the same job.
> 
> Can you or someone elaborate further how can I read the queryable state of a
> specific task from another task (e.g. map).
> 
> Best,
> Max
> 
> 
> 
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
> 


Mime
View raw message