giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maja Kabiljo" <majakabi...@fb.com>
Subject Re: Review Request 17336: Print job progress to command line
Date Tue, 28 Jan 2014 17:36:19 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17336/
-----------------------------------------------------------

(Updated Jan. 28, 2014, 5:36 p.m.)


Review request for giraph.


Changes
-------

Avery's comments


Bugs: GIRAPH-792
    https://issues.apache.org/jira/browse/GIRAPH-792


Repository: giraph-git


Description
-------

Currently we print nothing about job progress to command line. We should track which stage
are we in and how far in it are we.


Diffs (updated)
-----

  giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java 86823ed 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java c8b7d36 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 63f38df 
  giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java 1fe1d10 
  giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java f31d99e 
  giraph-core/src/main/java/org/apache/giraph/job/CombinedWorkerProgress.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/job/GiraphJob.java 40670bb 
  giraph-core/src/main/java/org/apache/giraph/job/HaltApplicationUtils.java 28b5781 
  giraph-core/src/main/java/org/apache/giraph/job/JobProgressTracker.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 78487ef 
  giraph-core/src/main/java/org/apache/giraph/utils/CounterUtils.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java bc29b03 
  giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 8ec0453

  giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java 01a6fc5

  giraph-core/src/main/java/org/apache/giraph/worker/WorkerProgress.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/worker/WorkerProgressWriter.java PRE-CREATION


Diff: https://reviews.apache.org/r/17336/diff/


Testing
-------

mvn clean verify

run on a cluster, checked there is no overhead (with 50 workers); sample output:
14/01/24 14:42:33 INFO job.JobProgressTracker: Data from 50 workers - Loading data: 315250000
vertices loaded, 0 vertex input splits loaded; 0 edges loaded, 0 edge input splits loaded
14/01/24 14:42:42 INFO job.JobProgressTracker: Data from 50 workers - Loading data: 441000000
vertices loaded, 64 vertex input splits loaded; 0 edges loaded, 0 edge input splits loaded
14/01/24 14:42:51 INFO job.JobProgressTracker: Data from 50 workers - Loading data: 494250000
vertices loaded, 234 vertex input splits loaded; 0 edges loaded, 0 edge input splits loaded
14/01/24 14:43:00 INFO job.JobProgressTracker: Data from 50 workers - Loading data: 498750000
vertices loaded, 247 vertex input splits loaded; 0 edges loaded, 0 edge input splits loaded
14/01/24 14:43:09 INFO job.JobProgressTracker: Data from 50 workers - Loading data: 499500000
vertices loaded, 249 vertex input splits loaded; 0 edges loaded, 0 edge input splits loaded
14/01/24 14:43:18 INFO job.JobProgressTracker: Data from 47 workers - Compute superstep 0:
6800000 out of 470000000 vertices computed; 0 out of 2350 partitions computed
14/01/24 14:43:27 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 0:
133200000 out of 500000000 vertices computed; 332 out of 2500 partitions computed
14/01/24 14:43:36 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 0:
304500000 out of 500000000 vertices computed; 1080 out of 2500 partitions computed
14/01/24 14:43:46 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 0:
467300000 out of 500000000 vertices computed; 2203 out of 2500 partitions computed
14/01/24 14:43:54 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 0:
500000000 out of 500000000 vertices computed; 2500 out of 2500 partitions computed
14/01/24 14:44:04 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 0:
500000000 out of 500000000 vertices computed; 2500 out of 2500 partitions computed
14/01/24 14:44:13 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 0:
500000000 out of 500000000 vertices computed; 2500 out of 2500 partitions computed
14/01/24 14:44:13 INFO mapred.ExpireTasks: Starting launching task sweep
14/01/24 14:44:21 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 0:
500000000 out of 500000000 vertices computed; 2500 out of 2500 partitions computed
14/01/24 14:44:30 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 0:
500000000 out of 500000000 vertices computed; 2500 out of 2500 partitions computed
14/01/24 14:44:39 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 0:
500000000 out of 500000000 vertices computed; 2500 out of 2500 partitions computed
14/01/24 14:44:48 INFO job.JobProgressTracker: Data from 13 workers - Compute superstep 1:
0 out of 130000000 vertices computed; 0 out of 650 partitions computed
14/01/24 14:44:57 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 1:
112600000 out of 500000000 vertices computed; 159 out of 2500 partitions computed
14/01/24 14:45:06 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 1:
268600000 out of 500000000 vertices computed; 1003 out of 2500 partitions computed
14/01/24 14:45:15 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 1:
418600000 out of 500000000 vertices computed; 2002 out of 2500 partitions computed
14/01/24 14:45:24 INFO job.JobProgressTracker: Data from 50 workers - Compute superstep 1:
499400000 out of 500000000 vertices computed; 2494 out of 2500 partitions computed


Thanks,

Maja Kabiljo


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message