hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Experiences with large hama cluster.
Date Thu, 03 Nov 2011 09:28:47 GMT
I tested Hama r.1197018 on 100 nodes cluster using 2 tasks per node.

The all the functions are works well, but task launching and finishing
are *very slow*.

[root@hnode-3-7 hama-trunk]# core/bin/start-bspd.sh
hnode-3-7: starting zookeeper, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-zookeeper-hnode-3-7.out
starting bspmaster, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-bspmaster-hnode-3-7.out
2011-11-03 16:29:14.382::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2011-11-03 16:29:14.430::INFO:  jetty-6.1.14
2011-11-03 16:29:14.684::INFO:  Started SelectChannelConnector@hnode-3-7:40013
hnode-3-7: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-7.out
hnode-3-8: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-8.out
hnode-3-10: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-10.out
hnode-3-9: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-9.out
hnode-3-12: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-12.out
hnode-3-13: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-13.out
hnode-3-11: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-11.out
hnode-3-15: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-15.out
hnode-3-16: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-16.out
hnode-3-18: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-18.out
hnode-3-14: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-14.out
hnode-3-17: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-17.out
hnode-3-20: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-20.out
hnode-3-21: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-21.out
hnode-3-19: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-19.out
hnode-3-22: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-22.out
hnode-3-23: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-23.out
hnode-3-24: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-24.out
hnode-3-28: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-28.out
hnode-3-25: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-25.out
hnode-3-27: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-27.out
hnode-3-26: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-26.out
hnode-3-30: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-30.out
hnode-3-29: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-29.out
hnode-3-31: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-31.out
hnode-3-32: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-32.out
hnode-3-33: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-33.out
hnode-3-35: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-35.out
hnode-3-34: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-34.out
hnode-3-37: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-37.out
hnode-3-38: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-38.out
hnode-3-36: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-36.out
hnode-3-39: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-39.out
hnode-3-40: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-40.out
hnode-3-42: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-42.out
hnode-3-43: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-43.out
hnode-3-45: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-45.out
hnode-3-41: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-41.out
hnode-3-44: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-44.out
hnode-3-46: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-46.out
hnode-3-47: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-47.out
hnode-3-49: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-49.out
hnode-3-50: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-50.out
hnode-3-51: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-51.out
hnode-3-48: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-48.out
hnode-3-55: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-55.out
hnode-3-53: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-53.out
hnode-3-52: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-52.out
hnode-3-56: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-56.out
hnode-3-54: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-54.out
hnode-3-60: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-60.out
hnode-3-59: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-59.out
hnode-3-58: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-58.out
hnode-3-57: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-57.out
hnode-3-62: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-62.out
hnode-3-63: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-63.out
hnode-3-64: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-64.out
hnode-3-65: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-65.out
hnode-3-67: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-67.out
hnode-3-70: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-70.out
hnode-3-61: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-61.out
hnode-3-66: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-66.out
hnode-3-68: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-68.out
hnode-3-72: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-72.out
hnode-3-69: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-69.out
hnode-3-71: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-71.out
hnode-3-75: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-75.out
hnode-3-74: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-74.out
hnode-3-73: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-73.out
hnode-3-77: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-77.out
hnode-3-76: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-76.out
hnode-3-79: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-79.out
hnode-3-78: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-78.out
hnode-3-81: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-81.out
hnode-3-80: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-80.out
hnode-3-82: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-82.out
hnode-3-83: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-83.out
hnode-3-85: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-85.out
hnode-3-86: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-86.out
hnode-3-84: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-84.out
hnode-3-87: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-87.out
hnode-3-88: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-88.out
hnode-3-89: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-89.out
hnode-3-90: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-90.out
hnode-3-91: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-91.out
hnode-3-95: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-95.out
hnode-3-93: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-93.out
hnode-3-94: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-94.out
hnode-3-92: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-92.out
hnode-3-96: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-96.out
hnode-3-97: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-97.out
hnode-3-98: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-98.out
hnode-3-99: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-99.out
hnode-3-102: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-102.out
hnode-3-100: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-100.out
hnode-3-104: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-104.out
hnode-3-101: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-101.out
hnode-3-105: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-105.out
hnode-3-106: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-106.out
hnode-3-103: starting groom, logging to
/usr/local/src/hama-trunk/core/bin/../logs/hama-root-groom-hnode-3-103.out
[root@hnode-3-7 hama-trunk]# ls
CHANGES.txt  core  DISCLAIMER.txt  examples  graph  lib  LICENSE.txt
NOTICE.txt  pom.xml  README.txt  src  target  yarn
[root@hnode-3-7 hama-trunk]# core/bin/hama jar
examples/target/hama-examples-0.4.0-incubating-SNAPSHOT.jar pi
11/11/03 16:36:45 INFO bsp.BSPJobClient: Running job: job_201111031629_0001
11/11/03 16:36:48 INFO bsp.BSPJobClient: Current supersteps number: 0
11/11/03 16:38:09 INFO bsp.BSPJobClient: Current supersteps number: 1
11/11/03 16:38:12 INFO bsp.BSPJobClient: The total number of supersteps: 1
Estimated value of PI is        3.1420720000000006
Job Finished in 90.868 seconds
[root@hnode-3-7 hama-trunk]# core/bin/hama jar
examples/target/hama-examples-0.4.0-incubating-SNAPSHOT.jar bench 16
1000 30
11/11/03 16:40:05 INFO bsp.BSPJobClient: Running job: job_201111031629_0002
11/11/03 16:40:08 INFO bsp.BSPJobClient: Current supersteps number: 0
11/11/03 16:41:47 INFO bsp.BSPJobClient: Current supersteps number: 1
11/11/03 16:41:50 INFO bsp.BSPJobClient: Current supersteps number: 3
11/11/03 16:41:53 INFO bsp.BSPJobClient: Current supersteps number: 5
11/11/03 16:41:59 INFO bsp.BSPJobClient: Current supersteps number: 7
11/11/03 16:42:02 INFO bsp.BSPJobClient: Current supersteps number: 10
11/11/03 16:42:05 INFO bsp.BSPJobClient: Current supersteps number: 14
11/11/03 16:42:08 INFO bsp.BSPJobClient: Current supersteps number: 19
11/11/03 16:42:11 INFO bsp.BSPJobClient: Current supersteps number: 24
11/11/03 16:42:14 INFO bsp.BSPJobClient: Current supersteps number: 28
11/11/03 16:42:17 INFO bsp.BSPJobClient: Current supersteps number: 30
11/11/03 16:42:20 INFO bsp.BSPJobClient: The total number of supersteps: 30
Job Finished in 136.781 seconds
[root@hnode-3-7 hama-trunk]# core/bin/hama jar
examples/target/hama-examples-0.4.0-incubating-SNAPSHOT.jar bench 16
10000 30
11/11/03 16:43:56 INFO bsp.BSPJobClient: Running job: job_201111031629_0003
11/11/03 16:43:59 INFO bsp.BSPJobClient: Current supersteps number: 0
11/11/03 16:45:42 INFO bsp.BSPJobClient: Current supersteps number: 1
11/11/03 16:45:48 INFO bsp.BSPJobClient: Current supersteps number: 2
11/11/03 16:45:54 INFO bsp.BSPJobClient: Current supersteps number: 3
11/11/03 16:45:57 INFO bsp.BSPJobClient: Current supersteps number: 4
11/11/03 16:46:03 INFO bsp.BSPJobClient: Current supersteps number: 5
11/11/03 16:46:06 INFO bsp.BSPJobClient: Current supersteps number: 6
11/11/03 16:46:09 INFO bsp.BSPJobClient: Current supersteps number: 7
11/11/03 16:46:12 INFO bsp.BSPJobClient: Current supersteps number: 8
11/11/03 16:46:15 INFO bsp.BSPJobClient: Current supersteps number: 9
11/11/03 16:46:18 INFO bsp.BSPJobClient: Current supersteps number: 10
11/11/03 16:46:21 INFO bsp.BSPJobClient: Current supersteps number: 11
11/11/03 16:46:24 INFO bsp.BSPJobClient: Current supersteps number: 12
11/11/03 16:46:27 INFO bsp.BSPJobClient: Current supersteps number: 13
11/11/03 16:46:30 INFO bsp.BSPJobClient: Current supersteps number: 14
11/11/03 16:46:33 INFO bsp.BSPJobClient: Current supersteps number: 15
11/11/03 16:46:36 INFO bsp.BSPJobClient: Current supersteps number: 16
11/11/03 16:46:39 INFO bsp.BSPJobClient: Current supersteps number: 17
11/11/03 16:46:42 INFO bsp.BSPJobClient: Current supersteps number: 19
11/11/03 16:46:45 INFO bsp.BSPJobClient: Current supersteps number: 20
11/11/03 16:46:51 INFO bsp.BSPJobClient: Current supersteps number: 21
11/11/03 16:46:54 INFO bsp.BSPJobClient: Current supersteps number: 22
11/11/03 16:46:57 INFO bsp.BSPJobClient: Current supersteps number: 23
11/11/03 16:47:00 INFO bsp.BSPJobClient: Current supersteps number: 24
11/11/03 16:47:03 INFO bsp.BSPJobClient: Current supersteps number: 25
11/11/03 16:47:06 INFO bsp.BSPJobClient: Current supersteps number: 26
11/11/03 16:47:09 INFO bsp.BSPJobClient: Current supersteps number: 27
11/11/03 16:47:12 INFO bsp.BSPJobClient: Current supersteps number: 28
11/11/03 16:47:15 INFO bsp.BSPJobClient: Current supersteps number: 29
11/11/03 16:47:18 INFO bsp.BSPJobClient: Current supersteps number: 30
11/11/03 16:47:24 INFO bsp.BSPJobClient: The total number of supersteps: 30
Job Finished in 209.516 seconds
[root@hnode-3-7 hama-trunk]# core/bin/hama jar
examples/target/hama-examples-0.4.0-incubating-SNAPSHOT.jar bench 16
100000 30
11/11/03 16:48:59 INFO bsp.BSPJobClient: Running job: job_201111031629_0004
11/11/03 16:49:02 INFO bsp.BSPJobClient: Current supersteps number: 0
11/11/03 16:50:50 INFO bsp.BSPJobClient: Current supersteps number: 1
11/11/03 16:51:08 INFO bsp.BSPJobClient: Current supersteps number: 2
11/11/03 16:51:26 INFO bsp.BSPJobClient: Current supersteps number: 3
11/11/03 16:51:44 INFO bsp.BSPJobClient: Current supersteps number: 4
11/11/03 16:51:59 INFO bsp.BSPJobClient: Current supersteps number: 5
11/11/03 16:52:17 INFO bsp.BSPJobClient: Current supersteps number: 6
11/11/03 16:52:39 INFO bsp.BSPJobClient: Current supersteps number: 7
11/11/03 16:52:54 INFO bsp.BSPJobClient: Current supersteps number: 8
11/11/03 16:53:12 INFO bsp.BSPJobClient: Current supersteps number: 9
11/11/03 16:53:39 INFO bsp.BSPJobClient: Current supersteps number: 10
11/11/03 16:53:57 INFO bsp.BSPJobClient: Current supersteps number: 11
11/11/03 16:54:12 INFO bsp.BSPJobClient: Current supersteps number: 12
11/11/03 16:54:30 INFO bsp.BSPJobClient: Current supersteps number: 13
11/11/03 16:54:48 INFO bsp.BSPJobClient: Current supersteps number: 14
11/11/03 16:55:03 INFO bsp.BSPJobClient: Current supersteps number: 15
11/11/03 16:55:18 INFO bsp.BSPJobClient: Current supersteps number: 16
11/11/03 16:55:36 INFO bsp.BSPJobClient: Current supersteps number: 17
11/11/03 16:55:51 INFO bsp.BSPJobClient: Current supersteps number: 18
11/11/03 16:56:09 INFO bsp.BSPJobClient: Current supersteps number: 19
11/11/03 16:56:24 INFO bsp.BSPJobClient: Current supersteps number: 20
11/11/03 16:56:42 INFO bsp.BSPJobClient: Current supersteps number: 21
11/11/03 16:56:57 INFO bsp.BSPJobClient: Current supersteps number: 22
11/11/03 16:57:15 INFO bsp.BSPJobClient: Current supersteps number: 23
11/11/03 16:57:30 INFO bsp.BSPJobClient: Current supersteps number: 24
11/11/03 16:57:48 INFO bsp.BSPJobClient: Current supersteps number: 25
11/11/03 16:58:09 INFO bsp.BSPJobClient: Current supersteps number: 26
11/11/03 16:58:24 INFO bsp.BSPJobClient: Current supersteps number: 27
11/11/03 16:58:39 INFO bsp.BSPJobClient: Current supersteps number: 28
11/11/03 16:58:57 INFO bsp.BSPJobClient: Current supersteps number: 29
11/11/03 16:59:12 INFO bsp.BSPJobClient: Current supersteps number: 30
11/11/03 16:59:36 INFO bsp.BSPJobClient: The total number of supersteps: 30
Job Finished in 639.003 seconds


-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message