pig-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Pig Wiki] Update of "PigMix" by AlanGates
Date Tue, 24 Feb 2009 19:18:34 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.

The following page has been changed by AlanGates:
http://wiki.apache.org/pig/PigMix

------------------------------------------------------------------------------
  
  == Runs ==
  
- == November 19, 2008 ==
- PigMix was run on a 25 machine hadoop cluster.  The cluster was running hadoop version 0.18.1.
 The tests were run against two
+ The following table includes runs done of the pig mix.  All of these runs have been done
on a cluster with 26 slaves plus one machine acting as the name node and job tracker.  The
cluster was running 
+ hadoop version 0.18.1.  (TODO:  Need to get specific hardware info on those machines). 

+ 
+ The tests were run against two
  versions of pig:  top of trunk, and top of types branch both as of Nov 21 2008.
  
- So far only the latency tests have been run, not the scalability tests.  Equivalent map
reduce programs have not yet been run.
- 
  The tests were run three times for each version and the results averaged.
  
- || Query || Top of Trunk || Top of Types Branch ||
- || L1 explode || 261 || 283 ||
- || L2 fr join || 1665 || 253 ||
- || L3 join || 1912 || 320 ||
- || L4 distinct agg || 254 || 193 ||
- || L5 anti-join || 1535 || 281 ||
- || L6 large group by key || 294 || 226 ||
- || L7 nested split || 243 || 204 ||
- || L8 group all || 462 ||  194 ||
- || L9 order by 1 field || 5294 || 867 ||
- || L10 order by multiple fields || 1403 || 565 ||
- || L11 distinct + union || 316 || 255 ||
- || L12 multi-store || fails || 781 ||
+ tot = top of trunk
+ totb = top of types branch
+ 
+ || Version                      || Map Reduce Java Code || tot 11/21/08 || totb 11/21/08
|| totb 1/20/09 ||
+ || Date Run                     || 11/22/08             || 11/21/08     || 11/21/08    
 || 1/20/09      ||
+ || L1 explode                   || 116                  || 261          || 283         
 || 218          ||
+ || L2 fr join                   ||  41                  || 1665         || 253         
 || 168          ||
+ || L3 join                      ||  97                  || 1912         || 320         
 || 258          ||
+ || L4 distinct agg              ||  68                  || 254          || 193         
 || 110          ||
+ || L5 anti-join                 ||  90                  || 1535         || 281         
 || 209          ||
+ || L6 large group by key        ||  61                  || 294          || 226         
 || 126          ||
+ || L7 nested split              ||  72                  || 243          || 204         
 || 107          ||
+ || L8 group all                 ||  56                  || 462          || 194         
 || 104          ||
+ || L9 order by 1 field          || 286                  || 5294         || 867         
 || 851          ||
+ || L10 order by multiple fields || 634                  || 1403         || 565         
 || 469          ||
+ || L11 distinct + union         || 120                  || 316          || 255         
 || 164          ||
+ || L12 multi-store              || 150                  || fails        || 781         
 || 499          ||
+ || Total time                   || 1791                 || 13638        || 4420        
 || 3284         ||
+ || Compared to hadoop           || 1.0                  || 7.6          || 2.5         
 || 1.8          ||
+ 
+ The totb run of 1/20/09 includes the change to make BufferedPositionedInputStream use a
buffer instead of relying on hadoop to buffer.
  
  == Features Tested ==
  Based on a sample of user queries, PigMix includes tests for the following features.

Mime
View raw message