Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Date: Fri, 1 Aug 2014 11:50:56 +0100 (IST)
From: Uli Bethke <uli.bethke@sonra.io>
Reply-To: Uli Bethke <uli.bethke@sonra.io>
To: user@hive.apache.org
Message-ID: 
 <414362556.104588.1406890256389.open-xchange@app03.ox.hosteurope.de>
In-Reply-To: 
 <CAORpBshDtkOzE55_CsCGdwMqVRA9xVONVBc+pMiQaLP-mwUnXA@mail.gmail.com>
References: 
 <1096099950.104351.1406889791005.open-xchange@app03.ox.hosteurope.de>
 <CAORpBshDtkOzE55_CsCGdwMqVRA9xVONVBc+pMiQaLP-mwUnXA@mail.gmail.com>
Subject: Re: Hive: Centralized HDFS Caching
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_104587_1369284670.1406890256329"
Importance: Medium

------=_Part_104587_1369284670.1406890256329
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

I am already using tez as the execution engine and used hdfs cacheadmin to pin a
file to memroy. However querying that file through Hive still goes to disk.

Any ideas?


> On 01 August 2014 at 11:46 Nitin Pawar <nitinpawar432@gmail.com> wrote:
> 
>  Please take a look at hive with tez as execution engine on hadoop 2.3.
> 
>  it may help you compare it with what you want to achieve
> 
> 
>  On Fri, Aug 1, 2014 at 4:13 PM, Uli Bethke <uli.bethke@sonra.io
> <mailto:uli.bethke@sonra.io> > wrote:
>    > >    Hi.
> > 
> >    in Hive can I make use of the centralized cache management introduced in
> > Hadoop 2.3 (
> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html)?
> > If not implemented yet, is this on the roadmap?
> > 
> >    My use case is that I want to pin a fact table that needs to be queried
> > frequently into memory.
> > 
> >    Impala already supports this as per the Cloudera documentation
> > http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_perf_hdfs_caching.html
> > 
> >    Thanks
> >    uli
> >  > 
> 
> 
>  --
>  Nitin Pawar
> 


------------------------------
Uli Bethke
Sonra. Unleash the Value of your Data.
Web: http://www.sonra.io
Skype: uli.bethke

ODI Training. Now available!
http://www.odi-training.com
Our ODI book on Amazon Kindle
http://amzn.to/1kDMFor
------=_Part_104587_1369284670.1406890256329
MIME-Version: 1.0
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
 
 </head><body style="">
 
  <div>
   I am already using tez as the execution engine and used hdfs cacheadmin to pin a file to memroy. However querying that file through Hive still goes to disk.
  </div> 
  <div>
   &#160;
  </div> 
  <div>
   Any ideas?
  </div> 
  <div>
   &#160;
  </div> 
  <blockquote style="position: relative; margin-left: 0px; padding-left: 10px; border-left: solid 1px blue;" type="cite">
   On 01 August 2014 at 11:46 Nitin Pawar &#60;nitinpawar432@gmail.com&#62; wrote:
   <br />
   <br /> 
   <div dir="ltr">
    Please take a look at hive with tez as execution engine on hadoop 2.3. 
    <div>
     &#160;
    </div> 
    <div>
     it may help you compare it with what you want to achieve&#160;
    </div> 
   </div> 
   <div class="gmail_extra">
    <br />
    <br /> 
    <div class="gmail_quote">
     On Fri, Aug 1, 2014 at 4:13 PM, Uli Bethke 
     <span>&#60;<a target="_blank" href="mailto:uli.bethke@sonra.io">uli.bethke@sonra.io</a>&#62;</span> wrote:
     <br /> 
     <blockquote style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">
      <span style="text-decoration: underline;"></span> 
      <div> 
       <div>
        Hi.
       </div> 
       <div>
        &#160;
       </div> 
       <div>
        in Hive can I make use of the centralized cache management introduced in Hadoop 2.3 (
        <a target="_blank" href="http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html</a>)? If not implemented yet, is this on the roadmap?
       </div> 
       <div>
        &#160;
       </div> 
       <div>
        My use case is that I want to pin a fact table that needs to be queried frequently into memory.
       </div> 
       <div>
        &#160;
       </div> 
       <div>
        Impala already supports this as per the Cloudera documentation 
        <a target="_blank" href="http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_perf_hdfs_caching.html">http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_perf_hdfs_caching.html</a>
       </div> 
       <div>
        &#160;
       </div> 
       <div>
        Thanks
       </div> 
       <div>
        uli
       </div> 
      </div> 
     </blockquote> 
    </div> 
    <br />
    <br clear="all" /> 
    <div>
     &#160;
    </div> -- 
    <br />Nitin Pawar
   </div> 
  </blockquote> 
  <div>
   <br />&#160;
  </div> 
  <div id="ox-signature">
   ------------------------------
   <br />Uli Bethke
   <br />Sonra. Unleash the Value of your Data.
   <br />Web: http://www.sonra.io
   <br />Skype: uli.bethke
   <br />
   <br />ODI Training. Now available!
   <br />http://www.odi-training.com
   <br />Our ODI book on Amazon Kindle
   <br />http://amzn.to/1kDMFor
  </div>
 
</body></html>
------=_Part_104587_1369284670.1406890256329--