Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <8E95E27A-00F2-4A41-9D76-E91CCEAFA459@gmail.com>
References: <8E95E27A-00F2-4A41-9D76-E91CCEAFA459@gmail.com>
Date: Fri, 7 Aug 2015 17:43:50 +0200
Message-ID: 
 <CAK5ODX7RY7bKXQxvejH09EeCpxGN+kY1+0Fzb8rtDCK31V7aWA@mail.gmail.com>
Subject: Re: extract workers' resources stats
From: Andra Lungu <lungu.andra@gmail.com>
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=047d7b62497c9eb551051cba8010

--047d7b62497c9eb551051cba8010
Content-Type: text/plain; charset=UTF-8

Hi,

You can use something like iostat to extract the CPU usage.
For instance, I call this script on the JM node:
#!/usr/bin/env bash

lines=`cat /home/andra.lungu/hostnames.txt | paste -d, -s`
echo "${lines}"
IFS="," read -ra hostnames <<<"${lines}"
for line in "${hostnames[@]}"; do
  ssh ${line} `iostat -xtc 5 3 > /home/andra.lungu/cpu_stats/${line}` &
done

And then I parse the individual output in the cpu_stats folder with python
to form charts.
To my knowledge. Flink does not do something like that for you, at least
not yet :)

Bottom line is that you can use linux commands: iostat, dstat, htop, etc.
Hope this helps!
Andra

On Fri, Aug 7, 2015 at 5:07 PM, Stefanos Antaris <antaris.stefanos@gmail.com
> wrote:

> Hi to all.
>
> I am working an a research project using flink and i would like to extract
> the CPU + RAM resources consumed on each worker in order to include the
> stats in my paper. Can anyone  advice me on how could i extract them?
>
> Thanks in advance.
> Stefanos

--047d7b62497c9eb551051cba8010
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div><div>Hi, <br><br></div>You c=
an use something like iostat to extract the CPU usage. <br></div>For instan=
ce, I call this script on the JM node:<br>#!/usr/bin/env bash<br><br>lines=
=3D`cat /home/andra.lungu/hostnames.txt | paste -d, -s`<br>echo &quot;${lin=
es}&quot;<br>IFS=3D&quot;,&quot; read -ra hostnames &lt;&lt;&lt;&quot;${lin=
es}&quot;<br>for line in &quot;${hostnames[@]}&quot;; do<br>=C2=A0 ssh ${li=
ne} `iostat -xtc 5 3 &gt; /home/andra.lungu/cpu_stats/${line}` &amp;<br>don=
e<br><br></div>And then I parse the individual output in the cpu_stats fold=
er with python to form charts. <br></div>To my knowledge. Flink does not do=
 something like that for you, at least not yet :)<br><br></div>Bottom line =
is that you can use linux commands: iostat, dstat, htop, etc.<br></div>Hope=
 this helps!<br></div>Andra<br></div><div class=3D"gmail_extra"><br><div cl=
ass=3D"gmail_quote">On Fri, Aug 7, 2015 at 5:07 PM, Stefanos Antaris <span =
dir=3D"ltr">&lt;<a href=3D"mailto:antaris.stefanos@gmail.com" target=3D"_bl=
ank">antaris.stefanos@gmail.com</a>&gt;</span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">Hi to all.<br>
<br>
I am working an a research project using flink and i would like to extract =
the CPU + RAM resources consumed on each worker in order to include the sta=
ts in my paper. Can anyone=C2=A0 advice me on how could i extract them?<br>
<br>
Thanks in advance.<br>
<span class=3D"HOEnZb"><font color=3D"#888888">Stefanos</font></span></bloc=
kquote></div><br></div>

--047d7b62497c9eb551051cba8010--