hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: hdsf block size
Date Thu, 17 Mar 2011 09:01:37 GMT
15 G single Gzip files? Consider block sizes in 0.5 GB+. But it also
depends on the processing slot-power you have. Higher blocks would
lead to higher usage of processing capacity, although with higher load
to the NameNode in maintaining lots of blocks (and replicas per) per
file.

On Thu, Mar 17, 2011 at 2:27 PM, Lior Schachter <liors@infolinks.com> wrote:
> Hi,
> We plan a 100T cluster with M/R jobs running on 15G gzip files.
> Should we configure HDFS block to be 128M or 256M.
>
> Thanks,
> Lior
>



-- 
Harsh J
http://harshj.com

Mime
View raw message