incubator-blur-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <>
Subject [Blur Wiki] Trivial Update of "MapReduce" by AaronMcCurry
Date Fri, 14 Jun 2013 01:41:18 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Blur Wiki" for change notification.

The "MapReduce" page has been changed by AaronMcCurry:

   * !BlurOutputFormat.setIndexLocally(Job,boolean)
    * Enabled by default, this will enable local indexing on the machine where the task is
running. Then when the !RecordWriter closes the index is copied to the remote destination
in HDFS.
   * !BlurOutputFormat.setMaxDocumentBufferSize(Job,int)
    * Sets the maximum number of documents that the buffer will hold in memory before overflowing
to disk. By default this is 1000 which will probably be very low for most systems.
   * !BlurOutputFormat.setOptimizeInFlight(Job,boolean)
    * Enabled by default, this will optimize the index while copying from the local index
to the remote destination in HDFS. Used in conjunction with the setIndexLocally.
   * !BlurOutputFormat.setReducerMultiplier(Job,int)
    * This will multiple the number of reducers for this job. For example if the table has
256 shards the normal number of reducers is 256. However if the reducer multiplier is set
to 4 then the number of reducers will be 1024 and each shard will get 4 new segments instead
of the normal 1.

View raw message