flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kruse, Sebastian" <Sebastian.Kr...@hpi.de>
Subject Job Profiling
Date Tue, 19 Aug 2014 09:08:51 GMT
Hi everyone,

I want to profile my flink jobs to find bottlenecks. I read the issue https://issues.apache.org/jira/browse/FLINK-964
and my question is whether there are currently ongoing efforts to bring the profiling data
to the web frontend.

Additionally, I was thinking of some kind of logical profiling, that measures the elements
(like tuples) being passed among the operators. That way one could better understand the properties
of intermediate data, e.g., join cardinalities. Plotting these data against a time axis, one
would come up with something like a data flow profile of the job. However, before engaging
in creating such profiles, I wanted to ask you if the system already keeps track of such data.
For instance, the job history graphs provide something similar, but the scheduling states
of tasks are not necessarily identical to the data flow through them.
I am happy for any comments!

Cheers,
Sebastian

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message