Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of jayunit100@gmail.com
 designates 209.85.210.54 as permitted sender)
Subject: Re: Distributed cache: how big is too big?
References: <BLU171-W104B9B90CEE4566EB53EC6C82C60@phx.gbl>
 <CA+qWQa0g=syWRPL+WX5ock5sY7kz=Z-jA1f0Y6YZSRCWsRj+Sw@mail.gmail.com>
From: Jay Vyas <jayunit100@gmail.com>
Content-Type: multipart/alternative;
	boundary=Apple-Mail-1E066DB3-A0C1-4C53-8F32-18D3F0ECAD4A
In-Reply-To: 
 <CA+qWQa0g=syWRPL+WX5ock5sY7kz=Z-jA1f0Y6YZSRCWsRj+Sw@mail.gmail.com>
Message-Id: <19410014-D07C-4A07-993C-B698D58A75AB@gmail.com>
Date: Tue, 9 Apr 2013 09:56:14 -0500
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (1.0)


--Apple-Mail-1E066DB3-A0C1-4C53-8F32-18D3F0ECAD4A
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable

Hmmm.. maybe im missing something.. but (@bjorn) Why would you use hdfs as a=
 replacement for the distributed cache?

After all - the distributed cache is just a file with replication over the w=
hole cluster, which isn't in hdfs.  Cant you Just make the cache size big an=
d store the file there?

What advantage is hdfs distribution of the file over all nodes  ?

On Apr 9, 2013, at 6:49 AM, Bjorn Jonsson <bjornjon@gmail.com> wrote:

> Put it once on hdfs with a replication factor equal to the number of DN. N=
o startup latency on job submission or max size and access it from anywhere w=
ith fs since it sticks around untill you replace it? Just a thought.
>=20
> On Apr 8, 2013 9:59 PM, "John Meza" <j_mezazap@hotmail.com> wrote:
>> I am researching a Hadoop solution for an existing application that requi=
res a directory structure full of data for processing.
>>=20
>> To make the Hadoop solution work I need to deploy the data directory to e=
ach DN when the job is executed.
>> I know this isn't new and commonly done with a Distributed Cache.
>>=20
>> Based on experience what are the common file sizes deployed in a Distribu=
ted Cache?=20
>> I know smaller is better, but how big is too big? the larger cache deploy=
ed I have read there will be startup latency. I also assume there are other f=
actors that play into this.
>>=20
>> I know that->Default local.cache.size=3D10Gb
>>=20
>> -Range of desirable sizes for Distributed Cache=3D 10Kb - 1Gb??
>> -Distributed Cache is normally not used if larger than =3D____?
>>=20
>> Another Option: Put the data directories on each DN and provide location t=
o TaskTracker?
>>=20
>> thanks
>> John

--Apple-Mail-1E066DB3-A0C1-4C53-8F32-18D3F0ECAD4A
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: 7bit

<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Hmmm.. maybe im missing something.. but (@bjorn) Why would you use hdfs as a replacement for the distributed cache?</div><div><br></div><div>After all - the distributed cache is just a file with replication over the whole cluster, which isn't in hdfs. &nbsp;Cant you Just make the cache size big and store the file there?<br><br>What advantage is hdfs distribution of the file over all nodes &nbsp;?</div><div><br>On Apr 9, 2013, at 6:49 AM, Bjorn Jonsson &lt;<a href="mailto:bjornjon@gmail.com">bjornjon@gmail.com</a>&gt; wrote:<br><br></div><blockquote type="cite"><div><p dir="ltr">Put it once on hdfs with a replication factor equal to the number of DN. No startup latency on job submission or max size and access it from anywhere with fs since it sticks around untill you replace it? Just a thought.</p>

<div class="gmail_quote">On Apr 8, 2013 9:59 PM, "John Meza" &lt;<a href="mailto:j_mezazap@hotmail.com">j_mezazap@hotmail.com</a>&gt; wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


<div><div dir="ltr">I am researching a Hadoop solution for an existing application that requires a directory structure full of data for processing.<div><br></div><div>To make the Hadoop solution work I need to deploy the data directory to each DN when the job is executed.</div>
<div>I know this isn't new and commonly done with a Distributed Cache.</div><div><br></div><div><b>Based on experience what are the common file sizes deployed in a Distributed Cache?</b>&nbsp;</div><div>I know smaller is better, but how big is too big? the larger cache deployed I have read there will be startup latency. I also assume there are other factors that play into this.</div>
<div><br></div><div><span style="color:rgb(80,80,80);font-family:monospace;font-size:13px;line-height:19px">I know that-&gt;Default local.cache.size=10Gb</span></div><div><br></div><div>-Range of desirable sizes for Distributed Cache= 10Kb - 1Gb??</div>
<div>-Distributed Cache is normally not used if larger than =____?</div><div><br></div><div><b>Another Option:</b> Put the data directories on each DN and provide location to TaskTracker?</div><div><br></div><div>thanks</div>
<div>John</div><div><br></div> 		 	   		  </div></div>
</blockquote></div>
</div></blockquote></body></html>
--Apple-Mail-1E066DB3-A0C1-4C53-8F32-18D3F0ECAD4A--