giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Harenberg <sdhar...@ncsu.edu>
Subject Re: Performance metrics / calling external scripts
Date Fri, 16 Jan 2015 20:40:13 GMT
Thanks for the info! I just looked into this and it is a nice feature.
However, I am ideally trying to make external calls to my script so that I
am obtaining metrics in a standard way while I compare multiple systems. In
the other systems, these phases were much more clear so the implementation
was quite simple.

After looking at the Giraph source further, I do think MasterThread.java is
the file I want. I now understand that superstep -1 is the input phase [1],
which can be benchmarked in MasterThread.java. Similarly, the entire vertex
computation phase can be benchmarked here too.

[1]
http://mail-archives.apache.org/mod_mbox/giraph-user/201404.mbox/%3CCAPOqDf2%2BzR3o%3DZGvnv5LYQ27gQnbOiUixg%3DxKCOfm1b3eTiN1Q%40mail.gmail.com%3E

On Wed, Jan 14, 2015 at 2:19 PM, Philipp Nolte <ptn@daslaboratorium.de>
wrote:

> In general, have you had a look at the giraph.metrics.enable option? It
> prints out metrics after each superstep to each worker's system.out log.
>
> -------- Urspr√ľngliche Nachricht --------
> Von: Steven Harenberg
> Datum:14.01.2015 19:55 (GMT+01:00)
> An: user
> Betreff: Performance metrics / calling external scripts
>
> Hi all,
>
> I am attempting to measure some performance metrics (such as runtime,
> memory usage, network communication, etc.) using an external bash script
> that grabs some machine stats.
>
> I am having difficulty figuring out where to externally call this script
> in Giraph. Particularly, I would like to call it at several key points in
> Giraph's execution, such as input/setup, beginning of computation, and
> output. The issue I am having is that I can't clearly figure out where to
> place the external calls because I can't figure out where these "phases"
> are actually happening in Giraph's source.
>
> I also have the added difficulty that I only want this external script to
> be called for each machine/worker not for each thread. Meaning, it should
> not be inside the vertex computation code, for example.
>
> Summary: my goal is to call an external script once per machine at
> the beginning of setup, computation (at/before superstep 0), and output.
>
>    1. Is this possible?
>    2. If so, could anyone please point me to where these phases are
>    happening that would work for making such an external call? I am guessing
>    this would be the MasterThread file, as this is where all the GiraphTimers
>    are happening.
>    3. Any general advice would be appreciated.
>
>
> Thanks and regards,
> Steve
>

Mime
View raw message