hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Panayotis Antonopoulos <antonopoulos...@hotmail.com>
Subject RE: Reducing Mapper InputSplit size
Date Tue, 07 Jun 2011 03:28:56 GMT

Hi Mark,

Check: http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html

I think that setMaxInputSplitSize(Job job,
                     long size)


will do what you need.

Regards,
P.A.

> Date: Mon, 6 Jun 2011 19:31:17 -0700
> Subject: Reducing Mapper InputSplit size
> From: markq2011@gmail.com
> To: common-user@hadoop.apache.org
> 
> Hi,
> 
> Does anyone have a way to reduce InputSplit size in general ?
> 
> By default, the minimum size chunk that map input should be split into is
> set to 0 (ie.mapred.min.split.size). Can I change dfs.block.size or some
> other configuration to reduce the split size and spawn many mappers?
> 
> Thanks,
> Mark
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message