hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 谢良 <xieli...@xiaomi.com>
Subject 答复: why my test result on dfs short circuit read is slower?
Date Sat, 16 Feb 2013 03:23:05 GMT
Hi Raymond,

did you enable security feature in your cluster?  there'll be no obvious benefit be found
if so.

Regards,
Liang
_______________________________________
发件人: Liu, Raymond [raymond.liu@intel.com]
发送时间: 2013年2月16日 11:10
收件人: user@hadoop.apache.org
主题: why my test result on dfs short circuit read is slower?

Hi

        I tried to use short circuit read to improve my hbase cluster MR scan performance.

        I have the following setting in hdfs-site.xml

        dfs.client.read.shortcircuit set to true
        dfs.block.local-path-access.user set to MR job runner.

        The cluster is 1+4 node and each data node have 16cpu/4HDD, with all hbase table major
compact thus all data is local.
        I have hoped that the short circuit read will improve the performance.

        While the test result is that with short circuit read enabled, the performance actually
dropped 10-15%. Say scan a 50G table cost around 100s instead of 90s.

        My hadoop version is 1.1.1, any idea on this? Thx!

Best Regards,
Raymond Liu
Mime
View raw message