flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: How to analyze space usage of Flink algorithms
Date Fri, 09 Dec 2016 13:51:34 GMT
We do not measure how much data we are spilling to disk.

On 09.12.2016 14:43, Fabian Hueske wrote:
> Hi,
>
> the heap mem usage should be available via Flink's metrics system.
> Not sure if that also captures spilled data. Chesnay (in CC) should 
> know that.
>
> If the spilled data is not available as a metric, you can try to write 
> a small script that monitors the directories to which Flink spills 
> (Config parameter: |taskmanager.tmp.dirs| [1]).
> The script would repeatedly list all files and keep for each file the 
> max size (files are deleted once the are not used anymore). This is 
> not super precise but might be good enough.
>
> Hope this helps,
> Fabian
>
> [1] 
> https://ci.apache.org/projects/flink/flink-docs-release-1.1/setup/config.html#jobmanager-amp-taskmanager
>
> 2016-12-09 14:12 GMT+01:00 otherwise777 <wouter@onzichtbaar.net 
> <mailto:wouter@onzichtbaar.net>>:
>
>     Currently i'm doing some analysis for some algorithms that i use
>     in Flink,
>     I'm interested in the Space and time it takes to execute them. For
>     the Time
>     i used getNetRuntime() in the executionenvironment, but I have no
>     idea how
>     to analyse the amount of space an algorithm uses.
>     Space can mean different things here, like Heap space, disk space,
>     overal
>     memory or allocated memory. I would like to analyze some of these.
>
>
>
>     --
>     View this message in context:
>     http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-space-usage-of-Flink-algorithms-tp10555.html
>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/How-to-analyze-space-usage-of-Flink-algorithms-tp10555.html>
>     Sent from the Apache Flink User Mailing List archive. mailing list
>     archive at Nabble.com.
>
>


Mime
View raw message