Return-Path: 
 <hdfs-user-return-2476-apmail-hadoop-hdfs-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 56B309840
	for <apmail-hadoop-hdfs-user-archive@minotaur.apache.org>;
 Thu, 26 Apr 2012 06:07:37 +0000 (UTC)
Received: (qmail 45822 invoked by uid 500); 26 Apr 2012 06:07:36 -0000
Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org
Received: (qmail 45385 invoked by uid 500); 26 Apr 2012 06:07:32 -0000
Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-user@hadoop.apache.org>
List-Id: <hdfs-user.hadoop.apache.org>
Reply-To: hdfs-user@hadoop.apache.org
Delivered-To: mailing list hdfs-user@hadoop.apache.org
Received: (qmail 45356 invoked by uid 99); 26 Apr 2012 06:07:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Apr 2012 06:07:32 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akshay.iiit@gmail.com
 designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-we0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Apr 2012 06:07:25 +0000
Received: by werc1 with SMTP id c1so740351wer.35
        for <hdfs-user@hadoop.apache.org>;
 Wed, 25 Apr 2012 23:07:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=rqhHSHaTdGNXg7f0WO4Gpvvx6yxGuNMOVK20Uq2VOEU=;
        b=XTWJT0kQLeMyhgZN+MFqUK86Wbeqjw06uIcKluoDKDSc10ys1Dg5MmRQSSNJINSW0y
         xv53hOgvw8WGn7A0l0n4h0hqGP979S0DqEefM0nGez20kvixzo2MybCdXy7KKQ9dYBS2
         3ssyYGgfAtKG7q7WyCD80eCVdVPRbNCJ7CxrnY4j30Z+UPisRd5L0GmWMJbBdB7TExxJ
         AUnmQH2XW/DMj0No6yvzwf9EtSMJSiw9thDL5pHjSeJWtmg3Lv1UxEAZ7z4Nzk/eWHBr
         h+9IbwO8Hsu2/TqTUmRgI0oHgE/6LzEsQKDr2aZ8VCXEzKCUyMBMnKlyDUZoFD1T8xYO
         o0HQ==
MIME-Version: 1.0
Received: by 10.180.82.136 with SMTP id i8mr13118254wiy.19.1335420424325; Wed,
 25 Apr 2012 23:07:04 -0700 (PDT)
Received: by 10.216.145.136 with HTTP; Wed, 25 Apr 2012 23:07:04 -0700 (PDT)
Date: Thu, 26 Apr 2012 02:07:04 -0400
Message-ID: 
 <CAAa7+2d014-0HrkE2J5XzFLZvT91Yr5=EpvSo=AiS1gPuSTN2w@mail.gmail.com>
Subject: file caching in HDFS ?
From: Akshay Singh <akshay.iiit@gmail.com>
To: hdfs-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=f46d04428c9c048c0d04be8ecca8
X-Virus-Checked: Checked by ClamAV on apache.org

--f46d04428c9c048c0d04be8ecca8
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I was looking for caching mechanisms in Hadoop, and was expecting
file/block caching on Datanodes for frequently accessed file-blocks.

As it seems, HDFS does not provide any caching below the file system
interface and utilizes DataNode's OS buffer cache for keeping frequently
accessed HDFS-file (stored as local OS files) in memory. Am i missing
anything ?

Also, is there any extension to HDFS which has implemented file caching at
DataNode level ? I understand that this another level of caching would
bring up issues like data-coherence, but I guess the performance gain may
be worth paying for this consistency overhead.

P.S. : I am looking for memory based cache on Datanodes, in case it was not
clear.

Thanks,
Akshay

--f46d04428c9c048c0d04be8ecca8
Content-Type: text/html; charset=ISO-8859-1

<div>Hi,</div><div><br></div><div>I was looking for  caching mechanisms 
in Hadoop, and was expecting file/block caching on Datanodes for 
frequently accessed file-blocks. <br></div><div><br></div><div>As it 
seems, HDFS does not provide any caching below the file system interface
 and utilizes DataNode&#39;s OS buffer cache for keeping frequently accessed
 HDFS-file (stored as local OS files) in memory. Am i missing anything ?</div><div><br></div><div>Also,
 is there any extension to HDFS which has implemented file caching at 
DataNode level ? I understand that this another level of caching would 
bring up issues like data-coherence, but I guess the performance gain 
may be worth paying for this consistency overhead.</div><div><br></div><div>P.S. : I am looking for memory based cache on Datanodes, in case it was not clear.
 <br></div><div><br></div><div>Thanks,</div>Akshay

--f46d04428c9c048c0d04be8ecca8--