hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jlei liu <liulei...@gmail.com>
Subject HDFS dfs.client.read.shortcircuit.skip.checksum
Date Sat, 15 Sep 2012 10:12:21 GMT
I use hadoop-0.20.2-cdh3u5 version, and config
dfs.client.read.shortcircuit=ture.


I use 10 threas to pread local file, the file is 700m and the OS cache the
file.

When I config the dfs.client.read.shortcircuit.skip.checksum=false, the TPS
is about 2000.

When I config the dfs.client.read.shortcircuit.skip.checksum=true, the TPS
is about 17000.

Why when don't read meta file, the performance improve 8 times?  If the
reason is there are two times seek,  I think whether we should save the
checksums to block file.


Thanks,

LiuLei

Mime
View raw message