hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: Caching in HDFS C API Client
Date Mon, 14 Jun 2010 21:08:03 GMT
Nice, thanks Brian!

On Jun 14, 2010, at 7:39 AM, Brian Bockelman wrote:

> Hey Owen, all,
>
> I find this one handy if you have root access:
>
> http://linux-mm.org/Drop_Caches
>
> echo 3 > /proc/sys/vm/drop_caches
>
> Drops the pagecache, dentries, and inodes.  Without this, you can  
> still get caching effects doing the normal "read and write large  
> files" if the linux pagecache outsmarts you (and I don't know about  
> you, but it often outsmarts me...).
>
> Brian
>
> On Jun 14, 2010, at 9:35 AM, Owen O'Malley wrote:
>
>> Indeed. On the terasort benchmark, I had to run intermediate jobs  
>> that
>> were larger than ram on the cluster to ensure that the data was not
>> coming from the file cache.
>>
>> -- Owen
>


Mime
View raw message