hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Trivial Update of "FAQ" by Arun C Murthy
Date Sun, 30 Sep 2007 12:49:04 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by Arun C Murthy:
http://wiki.apache.org/lucene-hadoop/FAQ

The comment on the change is:
Added a note on mapred.min.split.size for FAQ#10

------------------------------------------------------------------------------
  For this purpose one would need a 'non-splittable' [http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/FileInputFormat.html
FileInputFormat] i.e. an input-format which essentially tells the map-reduce framework that
it cannot be split-up and processed. To do this you need your particular input-format to return
'''false''' for the [http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/FileInputFormat.html#isSplitable(org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path)
isSplittable] call.
   
  E.g. '''org.apache.hadoop.mapred.Sort``Validator.Record``Stats``Checker.Non``Splitable``Sequence``File``Input``Format'''
in [http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/test/org/apache/hadoop/mapred/SortValidator.java
src/test/org/apache/hadoop/mapred/SortValidator.java]
-  
+ 
+ The other, quick-fix option, is to set [http://lucene.apache.org/hadoop/hadoop-default.html#mapred.min.split.size
mapred.min.split.size] to large enough value.
+ 
+ 
  [[BR]]
  [[Anchor(11)]]
  '''11. [#11 Why I do see broken images in jobdetails.jsp page?]'''

Mime
View raw message