giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nitay Joffe" <ni...@apache.org>
Subject Re: Review Request: GIRAPH-421: Aggregate metrics up to master
Date Sat, 17 Nov 2012 09:41:48 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8042/
-----------------------------------------------------------

(Updated Nov. 17, 2012, 9:41 a.m.)


Review request for giraph.


Description (updated)
-------

https://issues.apache.org/jira/browse/GIRAPH-421

The workers send their metrics to the master by writing them to the ZK node that is already
used for FinishedSuperstepStats.
The master reads each worker's metrics, aggregates, and prints an overall summary.

Worker:
https://gist.github.com/f0b6dce82a793173866a
Prints a summary at the end of each superstep. At the end of the job it does a raw dump of
all of the metrics. 

Master:
https://gist.github.com/99453c87bd33488588d4
At end of each superstep it gathers the metrics and computes mean, min, and max. Anything
else we should compute?
There is also support for the master to track and print its own metrics, but this is not really
used right now.

As a part of this diff I also cleaned some things up, namely:
- Just one option now to use, called giraph.metrics.enable, which toggles everything. When
false (default), no work is done. When enabled, every worker and the master tracks metrics.
They are aggregated and printed as in examples below.
- Changed metrics that are often small values like "time to first message" and "waiting time"
to be in microseconds instead of milliseconds.


Diffs
-----

  /trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/graph/BspService.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/graph/GraphMapper.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/metrics/AggregatedMetric.java PRE-CREATION

  /trunk/giraph/src/main/java/org/apache/giraph/metrics/AggregatedMetrics.java PRE-CREATION

  /trunk/giraph/src/main/java/org/apache/giraph/metrics/GiraphMetrics.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/metrics/GiraphMetricsRegistry.java 1409973

  /trunk/giraph/src/main/java/org/apache/giraph/metrics/MetricsRegistryDebugger.java PRE-CREATION

  /trunk/giraph/src/main/java/org/apache/giraph/metrics/SuperstepMetricsRegistry.java 1409973

  /trunk/giraph/src/main/java/org/apache/giraph/metrics/ValueWithHostname.java PRE-CREATION

  /trunk/giraph/src/main/java/org/apache/giraph/metrics/WorkerSuperstepMetrics.java PRE-CREATION

  /trunk/giraph/src/main/java/org/apache/giraph/utils/FakeTime.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/utils/SystemTime.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/utils/Time.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/utils/Times.java 1409973 
  /trunk/giraph/src/main/java/org/apache/giraph/utils/WritableUtils.java 1409973 

Diff: https://reviews.apache.org/r/8042/diff/


Testing
-------


Thanks,

Nitay Joffe


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message