hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Trivial Update of "DataProcessingBenchmarks" by udanax
Date Tue, 08 Jan 2008 08:21:49 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by udanax:
http://wiki.apache.org/lucene-hadoop/DataProcessingBenchmarks

New page:
== Data Processing Benchmarks ==

SQL > select ipaddress, count(*) from access_log group by ipaddress order by count(*) desc
limit 0,100;
[[BR]]''σ ,,count. ipaddress,, (τ ,,count,, (γ ,,count(ipaddress). ipaddress,, (access_log)))''

||<bgcolor="#E5E5E5">||<bgcolor="#E5E5E5">!MySql ||<bgcolor="#E5E5E5">Hadoop||
||<bgcolor="#E5E5E5">Index ||index ||non-index||
||<bgcolor="#E5E5E5">Machine ||1 ||1000||
||<bgcolor="#E5E5E5">Rows ||3,700,000 ||49,324,734||
||<bgcolor="#E5E5E5">Results ||100 ||100||
||<bgcolor="#E5E5E5">Time  ||3.715 sec ||112.03 sec||

{{{
bash# ./bin/hadoop jar ./log_examples.jar loganalysis -m 1000 -r 1000 udanax/logfiles udanax/rank
100

* Connecting Time Count Map/Reduce Job start.
----------------------------------------------------------------------
08/01/08 16:13:46 INFO mapred.FileInputFormat: Total input paths to process : 18
08/01/08 16:13:47 INFO mapred.JobClient: Running job: job_200801081529_0005
08/01/08 16:13:48 INFO mapred.JobClient:  map 0% reduce 0%
08/01/08 16:13:54 INFO mapred.JobClient:  map 1% reduce 0%
08/01/08 16:13:56 INFO mapred.JobClient:  map 2% reduce 0%
08/01/08 16:14:01 INFO mapred.JobClient:  map 6% reduce 0%
08/01/08 16:14:03 INFO mapred.JobClient:  map 8% reduce 0%
08/01/08 16:14:05 INFO mapred.JobClient:  map 10% reduce 0%
08/01/08 16:14:07 INFO mapred.JobClient:  map 23% reduce 0%
08/01/08 16:14:08 INFO mapred.JobClient:  map 24% reduce 0%
08/01/08 16:14:11 INFO mapred.JobClient:  map 28% reduce 0%
08/01/08 16:14:13 INFO mapred.JobClient:  map 29% reduce 0%
08/01/08 16:14:15 INFO mapred.JobClient:  map 31% reduce 0%
08/01/08 16:14:16 INFO mapred.JobClient:  map 33% reduce 0%
08/01/08 16:14:17 INFO mapred.JobClient:  map 39% reduce 1%
08/01/08 16:14:18 INFO mapred.JobClient:  map 40% reduce 1%
08/01/08 16:14:21 INFO mapred.JobClient:  map 43% reduce 1%
08/01/08 16:14:23 INFO mapred.JobClient:  map 44% reduce 2%
08/01/08 16:14:24 INFO mapred.JobClient:  map 45% reduce 2%
08/01/08 16:14:25 INFO mapred.JobClient:  map 46% reduce 2%
08/01/08 16:14:26 INFO mapred.JobClient:  map 48% reduce 3%
08/01/08 16:14:27 INFO mapred.JobClient:  map 52% reduce 4%
08/01/08 16:14:30 INFO mapred.JobClient:  map 53% reduce 4%
08/01/08 16:14:31 INFO mapred.JobClient:  map 54% reduce 5%
08/01/08 16:14:33 INFO mapred.JobClient:  map 55% reduce 5%
08/01/08 16:14:34 INFO mapred.JobClient:  map 56% reduce 5%
08/01/08 16:14:35 INFO mapred.JobClient:  map 57% reduce 5%
08/01/08 16:14:36 INFO mapred.JobClient:  map 61% reduce 7%
08/01/08 16:14:37 INFO mapred.JobClient:  map 63% reduce 7%
08/01/08 16:14:41 INFO mapred.JobClient:  map 64% reduce 7%
08/01/08 16:14:43 INFO mapred.JobClient:  map 65% reduce 7%
08/01/08 16:14:45 INFO mapred.JobClient:  map 67% reduce 8%
08/01/08 16:14:47 INFO mapred.JobClient:  map 72% reduce 8%
08/01/08 16:14:48 INFO mapred.JobClient:  map 73% reduce 9%
08/01/08 16:14:52 INFO mapred.JobClient:  map 74% reduce 9%
08/01/08 16:14:55 INFO mapred.JobClient:  map 76% reduce 10%
08/01/08 16:14:57 INFO mapred.JobClient:  map 80% reduce 11%
08/01/08 16:14:59 INFO mapred.JobClient:  map 81% reduce 11%
08/01/08 16:15:01 INFO mapred.JobClient:  map 82% reduce 11%
08/01/08 16:15:02 INFO mapred.JobClient:  map 83% reduce 11%
08/01/08 16:15:05 INFO mapred.JobClient:  map 86% reduce 12%
08/01/08 16:15:06 INFO mapred.JobClient:  map 87% reduce 12%
08/01/08 16:15:07 INFO mapred.JobClient:  map 88% reduce 13%
08/01/08 16:15:09 INFO mapred.JobClient:  map 89% reduce 13%
08/01/08 16:15:10 INFO mapred.JobClient:  map 89% reduce 14%
08/01/08 16:15:11 INFO mapred.JobClient:  map 90% reduce 14%
08/01/08 16:15:12 INFO mapred.JobClient:  map 91% reduce 14%
08/01/08 16:15:14 INFO mapred.JobClient:  map 92% reduce 14%
08/01/08 16:15:15 INFO mapred.JobClient:  map 94% reduce 15%
08/01/08 16:15:16 INFO mapred.JobClient:  map 95% reduce 16%
08/01/08 16:15:17 INFO mapred.JobClient:  map 96% reduce 16%
08/01/08 16:15:18 INFO mapred.JobClient:  map 97% reduce 16%
08/01/08 16:15:20 INFO mapred.JobClient:  map 98% reduce 16%
08/01/08 16:15:22 INFO mapred.JobClient:  map 98% reduce 17%
08/01/08 16:15:24 INFO mapred.JobClient:  map 100% reduce 17%
08/01/08 16:15:26 INFO mapred.JobClient:  map 100% reduce 18%
08/01/08 16:15:27 INFO mapred.JobClient:  map 100% reduce 31%
08/01/08 16:15:28 INFO mapred.JobClient:  map 100% reduce 36%
08/01/08 16:15:29 INFO mapred.JobClient:  map 100% reduce 45%
08/01/08 16:15:30 INFO mapred.JobClient:  map 100% reduce 74%
08/01/08 16:15:31 INFO mapred.JobClient:  map 100% reduce 81%
08/01/08 16:15:32 INFO mapred.JobClient:  map 100% reduce 84%
08/01/08 16:15:33 INFO mapred.JobClient:  map 100% reduce 92%
08/01/08 16:15:34 INFO mapred.JobClient:  map 100% reduce 100%
08/01/08 16:15:35 INFO mapred.JobClient: Job complete: job_200801081529_0005
08/01/08 16:15:35 INFO mapred.JobClient: Counters: 11
08/01/08 16:15:35 INFO mapred.JobClient:   Job Counters
08/01/08 16:15:35 INFO mapred.JobClient:     Launched map tasks=1355
08/01/08 16:15:35 INFO mapred.JobClient:     Launched reduce tasks=1457
08/01/08 16:15:35 INFO mapred.JobClient:   Map-Reduce Framework
08/01/08 16:15:35 INFO mapred.JobClient:     Map input records=49324734
08/01/08 16:15:35 INFO mapred.JobClient:     Map output records=49324721
08/01/08 16:15:35 INFO mapred.JobClient:     Map input bytes=8551673779
08/01/08 16:15:35 INFO mapred.JobClient:     Map output bytes=790763358
08/01/08 16:15:35 INFO mapred.JobClient:     Combine input records=49324734
08/01/08 16:15:35 INFO mapred.JobClient:     Combine output records=705771
08/01/08 16:15:35 INFO mapred.JobClient:     Reduce input groups=201330
08/01/08 16:15:35 INFO mapred.JobClient:     Reduce input records=705771
08/01/08 16:15:35 INFO mapred.JobClient:     Reduce output records=201330

* Sort by Connection Time Count Map/Reduce Job start.
----------------------------------------------------------------------
08/01/08 16:15:35 INFO mapred.FileInputFormat: Total input paths to process : 100
08/01/08 16:15:36 INFO mapred.JobClient: Running job: job_200801081529_0006
08/01/08 16:15:37 INFO mapred.JobClient:  map 0% reduce 0%
08/01/08 16:15:40 INFO mapred.JobClient:  map 10% reduce 0%
08/01/08 16:15:41 INFO mapred.JobClient:  map 42% reduce 0%
08/01/08 16:15:42 INFO mapred.JobClient:  map 91% reduce 0%
08/01/08 16:15:43 INFO mapred.JobClient:  map 100% reduce 0%
08/01/08 16:17:54 INFO mapred.JobClient:  map 100% reduce 6%
08/01/08 16:20:54 INFO mapred.JobClient:  map 100% reduce 17%
08/01/08 16:22:41 INFO mapred.JobClient:  map 100% reduce 29%
08/01/08 16:25:52 INFO mapred.JobClient:  map 100% reduce 37%
08/01/08 16:27:44 INFO mapred.JobClient:  map 100% reduce 51%
08/01/08 16:28:12 INFO mapred.JobClient:  map 100% reduce 69%
08/01/08 16:30:35 INFO mapred.JobClient:  map 100% reduce 82%
08/01/08 16:32:25 INFO mapred.JobClient:  map 100% reduce 99%
08/01/08 16:33:54 INFO mapred.JobClient:  map 100% reduce 100%
08/01/08 16:33:55 INFO mapred.JobClient: Job complete: job_200801081529_0006
08/01/08 16:33:11 INFO mapred.JobClient: Counters: 11
08/01/08 16:33:55 INFO mapred.JobClient:   Job Counters
08/01/08 16:33:55 INFO mapred.JobClient:     Launched map tasks=1080
08/01/08 16:33:55 INFO mapred.JobClient:     Launched reduce tasks=1
08/01/08 16:33:55 INFO mapred.JobClient:   Map-Reduce Framework
08/01/08 16:33:55 INFO mapred.JobClient:     Map input records=201330
08/01/08 16:33:55 INFO mapred.JobClient:     Map output records=201330
08/01/08 16:33:55 INFO mapred.JobClient:     Map input bytes=5080608
08/01/08 16:33:55 INFO mapred.JobClient:     Map output bytes=5108994
08/01/08 16:33:55 INFO mapred.JobClient:     Combine input records=201330
08/01/08 16:33:55 INFO mapred.JobClient:     Combine output records=8406
08/01/08 16:33:55 INFO mapred.JobClient:     Reduce input groups=19270
08/01/08 16:33:55 INFO mapred.JobClient:     Reduce input records=84069
08/01/08 16:33:55 INFO mapred.JobClient:     Reduce output records=200

------------------------------------
* Top 100 connector list :
+--------------+-------------------+
| Count        | Ip Address        |
+--------------+-------------------+
| 374932      | 121.165.51.179    |
| 357615      | 121.150.85.42     |
| 304878      | 211.204.83.50     |
| 274461      | 10.8.107.219      |
| 264475      | 222.238.215.220   |
| 246650      | 123.254.226.176   |
| 242124      | 218.50.17.33      |
| 229223      | 116.34.227.130    |
| 222771      | 61.98.219.133     |
| 196677      | 116.122.252.19    |
| 186095      | 124.46.159.54     |
| 181853      | 211.172.61.217    |
| 178051      | 123.214.135.227   |
| 177545      | 222.118.198.139   |
| 175318      | 121.190.177.28    |
| 174289      | 211.107.182.103   |
| 166152      | 123.212.206.195   |
| 165495      | 218.50.28.205     |
| 164378      | 59.3.99.227       |
| 160612      | 121.144.99.83     |
| 158529      | 219.252.85.66     |
| 153793      | 121.138.193.64    |
| 151995      | 121.145.170.142   |
| 146165      | 211.245.129.176   |
| 145766      | 124.50.192.5      |
| 141955      | 124.49.112.28     |
| 139306      | 74.6.22.134       |
| 138591      | 121.146.230.146   |
| 136510      | 222.104.236.39    |
| 135713      | 222.119.64.36     |
| 135395      | 211.214.79.219    |
| 131614      | 124.50.39.137     |
| 130607      | 211.203.169.195   |
| 129805      | 121.184.49.152    |
| 128691      | 125.138.220.135   |
| 127131      | 222.109.169.162   |
| 126593      | 58.235.62.31      |
| 126572      | 211.209.220.182   |
| 124235      | 121.166.8.143     |
| 120835      | 210.94.77.21      |
| 119125      | 121.132.10.153    |
| 118614      | 222.237.43.156    |
| 116033      | 222.232.155.39    |
| 114882      | 124.60.159.9      |
| 114030      | 218.238.122.191   |
| 112950      | 121.146.203.67    |
| 110689      | 222.232.72.138    |
| 109602      | 59.27.78.202      |
| 107644      | 121.171.103.242   |
| 107455      | 221.150.183.226   |
| 107152      | 218.55.200.198    |
| 105625      | 122.36.134.52     |
| 105108      | 58.142.103.228    |
| 103540      | 121.189.210.44    |
| 103379      | 125.191.119.191   |
| 103302      | 121.135.12.61     |
| 103068      | 59.150.103.15     |
| 102877      | 211.33.116.85     |
| 102724      | 65.55.213.39      |
| 102605      | 121.131.89.5      |
| 102203      | 221.150.198.209   |
| 102059      | 125.190.91.14     |
| 101727      | 125.187.35.180    |
| 101624      | 58.228.81.115     |
| 100364      | 203.247.80.42     |
| 98559       | 121.188.233.2     |
| 97796       | 124.51.131.138    |
| 95466       | 222.239.59.19     |
| 94386       | 122.128.194.104   |
| 89315       | 61.252.130.101    |
| 89205       | 116.37.203.91     |
| 87911       | 125.187.54.146    |
| 87851       | 125.142.135.237   |
| 86262       | 121.165.66.27     |
| 85723       | 58.234.15.202     |
| 85086       | 121.165.73.254    |
| 84824       | 121.53.95.93      |
| 84762       | 211.36.65.234     |
| 84409       | 58.121.165.252    |
| 84329       | 168.131.153.175   |
| 82710       | 121.162.28.131    |
| 82041       | 121.139.151.137   |
| 81887       | 124.111.242.97    |
| 81578       | 125.178.79.218    |
| 80827       | 124.62.208.102    |
| 79866       | 211.175.253.135   |
| 79226       | 202.30.106.20     |
| 78535       | 58.122.138.79     |
| 77824       | 221.161.127.18    |
| 77141       | 211.59.140.121    |
| 76623       | 122.35.247.26     |
| 73814       | 210.183.41.161    |
| 73328       | 125.182.29.4      |
| 73220       | 218.236.193.29    |
| 72808       | 61.101.164.172    |
| 72413       | 58.120.250.151    |
| 72154       | 211.210.164.215   |
| 72083       | 122.44.149.231    |
| 71646       | 124.49.150.145    |
| 70915       | 211.48.70.247     |
+--------------+-------------------+
Processing time : 112.03 sec
}}}

Mime
View raw message