hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From elton sky <eltonsky9...@gmail.com>
Subject increase BytesPerChecksum decrease write performance??
Date Sat, 09 Oct 2010 04:42:30 GMT

I was benchmarking write/read of HDFS.

I changed the chunksize, i.e. bytesPerChecksum or bpc, and create a 1G file
with 128MB block size. The bpc I used: 512B, 32KB, 64KB, 256KB, 512KB, 2MB,

The result surprised me. The performance for 512B, 32KB, 64KB are quite
similar, and then, as the increase of the bpc size the throughput decreases.
And comparing 512B to 8MB, there's a 40% to 50% difference in throughput.

Is there any idea for this?

View raw message