flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kruse, Sebastian" <Sebastian.Kr...@hpi.de>
Subject RE: Job Profiling
Date Fri, 22 Aug 2014 13:12:59 GMT
Thanks, I will have a look at it and see if I can figure out how to do that.

-----Original Message-----
From: ewenstephan@gmail.com [mailto:ewenstephan@gmail.com] On Behalf Of Stephan Ewen
Sent: Dienstag, 19. August 2014 16:07
To: dev@flink.incubator.apache.org
Subject: Re: Job Profiling

Hi Sebastian!

There is some profiling code that was used by previous versions of Flink (Stratosphere). The
profiling works, but there is currently nothing that displays the profiling data.

It would be a great addition to integrate displaying the profiling code in the web frontend,
or making it available for download.

Have a look at those classes here:
 - JobManager side :
https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/profiling/impl/JobManagerProfilerImpl.java
 - TaskManager sied :
https://github.com/apache/incubator-flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/profiling/impl/TaskManagerProfilerImpl.java

Daniel Warneke authored those, maybe he can chime in and give a few pointers

Greetings,
Stephan




On Tue, Aug 19, 2014 at 11:08 AM, Kruse, Sebastian <Sebastian.Kruse@hpi.de>
wrote:

> Hi everyone,
>
> I want to profile my flink jobs to find bottlenecks. I read the issue
> https://issues.apache.org/jira/browse/FLINK-964 and my question is 
> whether there are currently ongoing efforts to bring the profiling 
> data to the web frontend.
>
> Additionally, I was thinking of some kind of logical profiling, that 
> measures the elements (like tuples) being passed among the operators. 
> That way one could better understand the properties of intermediate 
> data, e.g., join cardinalities. Plotting these data against a time 
> axis, one would come up with something like a data flow profile of the 
> job. However, before engaging in creating such profiles, I wanted to 
> ask you if the system already keeps track of such data. For instance, 
> the job history graphs provide something similar, but the scheduling 
> states of tasks are not necessarily identical to the data flow through them.
> I am happy for any comments!
>
> Cheers,
> Sebastian
>
Mime
View raw message