hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Update of "FAQ" by NigelDaley
Date Wed, 13 Jun 2007 00:39:49 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by NigelDaley:
http://wiki.apache.org/lucene-hadoop/FAQ

The comment on the change is:
added 900 node config values

------------------------------------------------------------------------------
  
  == 3. How well does Hadoop scale? ==
  
- Hadoop has been demonstrated on clusters of up to 2000 nodes.  Sort performance is good
(sorting 9TB of data on 900 nodes takes around 4 hours) and [attachment:sort900-20070607.png
improving].
+ Hadoop has been demonstrated on clusters of up to 2000 nodes.  Sort performance on 900 nodes
is good (sorting 9TB of data on 900 nodes takes around 4 hours) and [attachment:sort900-20070607.png
improving] using these non-default configuration values:
+ 
+   * `dfs.block.size = 134217728`
+   * `dfs.namenode.handler.count = 40`
+   * `mapred.speculative.execution = true`
+   * `mapred.reduce.parallel.copies = 20`
+   * `mapred.tasktracker.tasks.maximum = 2`
+   * `mapred.child.java.opts = -Xmx512m`
+   * `fs.inmemory.size.mb = 200`
+   * `io.sort.factor = 100`
+   * `io.sort.mb = 200`
+   * `io.file.buffer.size = 131072`
  
  == 4. Do I have to write my application in Java? ==
  

Mime
View raw message