hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshay Singh <akshay.i...@gmail.com>
Subject file caching in HDFS ?
Date Thu, 26 Apr 2012 06:07:04 GMT
Hi,

I was looking for caching mechanisms in Hadoop, and was expecting
file/block caching on Datanodes for frequently accessed file-blocks.

As it seems, HDFS does not provide any caching below the file system
interface and utilizes DataNode's OS buffer cache for keeping frequently
accessed HDFS-file (stored as local OS files) in memory. Am i missing
anything ?

Also, is there any extension to HDFS which has implemented file caching at
DataNode level ? I understand that this another level of caching would
bring up issues like data-coherence, but I guess the performance gain may
be worth paying for this consistency overhead.

P.S. : I am looking for memory based cache on Datanodes, in case it was not
clear.

Thanks,
Akshay

Mime
View raw message