Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of bjornjon@gmail.com designates
 209.85.210.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BLU171-W104B9B90CEE4566EB53EC6C82C60@phx.gbl>
References: <BLU171-W104B9B90CEE4566EB53EC6C82C60@phx.gbl>
Date: Tue, 9 Apr 2013 04:49:17 -0700
Message-ID: 
 <CA+qWQa0g=syWRPL+WX5ock5sY7kz=Z-jA1f0Y6YZSRCWsRj+Sw@mail.gmail.com>
Subject: Re: Distributed cache: how big is too big?
From: Bjorn Jonsson <bjornjon@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b418791ad587204d9ec24a1

--047d7b418791ad587204d9ec24a1
Content-Type: text/plain; charset=ISO-8859-1

Put it once on hdfs with a replication factor equal to the number of DN. No
startup latency on job submission or max size and access it from anywhere
with fs since it sticks around untill you replace it? Just a thought.
On Apr 8, 2013 9:59 PM, "John Meza" <j_mezazap@hotmail.com> wrote:

> I am researching a Hadoop solution for an existing application that
> requires a directory structure full of data for processing.
>
> To make the Hadoop solution work I need to deploy the data directory to
> each DN when the job is executed.
> I know this isn't new and commonly done with a Distributed Cache.
>
> *Based on experience what are the common file sizes deployed in a
> Distributed Cache?*
> I know smaller is better, but how big is too big? the larger cache
> deployed I have read there will be startup latency. I also assume there are
> other factors that play into this.
>
> I know that->Default local.cache.size=10Gb
>
> -Range of desirable sizes for Distributed Cache= 10Kb - 1Gb??
> -Distributed Cache is normally not used if larger than =____?
>
> *Another Option:* Put the data directories on each DN and provide
> location to TaskTracker?
>
> thanks
> John
>
>

--047d7b418791ad587204d9ec24a1
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">Put it once on hdfs with a replication factor equal to the n=
umber of DN. No startup latency on job submission or max size and access it=
 from anywhere with fs since it sticks around untill you replace it? Just a=
 thought.</p>

<div class=3D"gmail_quote">On Apr 8, 2013 9:59 PM, &quot;John Meza&quot; &l=
t;<a href=3D"mailto:j_mezazap@hotmail.com">j_mezazap@hotmail.com</a>&gt; wr=
ote:<br type=3D"attribution"><blockquote class=3D"gmail_quote" style=3D"mar=
gin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


<div><div dir=3D"ltr">I am researching a Hadoop solution for an existing ap=
plication that requires a directory structure full of data for processing.<=
div><br></div><div>To make the Hadoop solution work I need to deploy the da=
ta directory to each DN when the job is executed.</div>
<div>I know this isn&#39;t new and commonly done with a Distributed Cache.<=
/div><div><br></div><div><b>Based on experience what are the common file si=
zes deployed in a Distributed Cache?</b>=A0</div><div>I know smaller is bet=
ter, but how big is too big? the larger cache deployed I have read there wi=
ll be startup latency. I also assume there are other factors that play into=
 this.</div>
<div><br></div><div><span style=3D"color:rgb(80,80,80);font-family:monospac=
e;font-size:13px;line-height:19px">I know that-&gt;Default local.cache.size=
=3D10Gb</span></div><div><br></div><div>-Range of desirable sizes for Distr=
ibuted Cache=3D 10Kb - 1Gb??</div>
<div>-Distributed Cache is normally not used if larger than =3D____?</div><=
div><br></div><div><b>Another Option:</b> Put the data directories on each =
DN and provide location to TaskTracker?</div><div><br></div><div>thanks</di=
v>
<div>John</div><div><br></div> 		 	   		  </div></div>
</blockquote></div>

--047d7b418791ad587204d9ec24a1--