Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of shahab.yunus@gmail.com
 designates 209.85.217.178 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANS3Q-kye2=z-cWz-Pw72R+HnTeAXmWtpgC53H2NHQwVtZq3mQ@mail.gmail.com>
References: 
 <CANS3Q-kye2=z-cWz-Pw72R+HnTeAXmWtpgC53H2NHQwVtZq3mQ@mail.gmail.com>
Date: Thu, 11 Dec 2014 23:25:01 -0500
Message-ID: 
 <CAEo-6+T4b0WWgOqsVAcDzS7LxwaF-zAeLGTmUcveCxZE5pQypw@mail.gmail.com>
Subject: Re: DistributedCache
From: Shahab Yunus <shahab.yunus@gmail.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a11c3d510baf4c50509fd46f8

--001a11c3d510baf4c50509fd46f8
Content-Type: text/plain; charset=UTF-8

Look at this thread. It has alternatives to DistributedCache.
http://stackoverflow.com/questions/21239722/hadoop-distributedcache-is-deprecated-what-is-the-preferred-api

Basically you can use the new method job.addCacheFiles to pass on stuff to
the individual tasks.

Regards,
Shahab

On Thu, Dec 11, 2014 at 9:07 PM, Srinivas Chamarthi <
srinivas.chamarthi@gmail.com> wrote:
>
> Hi,
>
> I want to cache map/reducer temporary output files so that I can compare
> two map results coming from two different nodes to verify the integrity
> check.
>
> I am simulating this use case with speculative execution by rescheduling
> the first task as soon as it is started and running.
>
> Now I want to compare output files coming from speculative attempt and
> prior attempt so that I can calculate the credit scoring of each node.
>
> I want to use DistributedCache to cache the local file system files in
> CommitPending stage from TaskImpl. But the DistributedCache is actually
> deprecated. is there any other way I can do this ?
>
> I think I can use HDFS to save the temporary output files so that other
> nodes can see it ? but is there any in-memory solution I can use ?
>
> any pointers are greatly appreciated.
>
> thx & rgds,
> srinivas chamarthi
>

--001a11c3d510baf4c50509fd46f8
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Look at this thread. It has alternatives to DistributedCac=
he.<div><a href=3D"http://stackoverflow.com/questions/21239722/hadoop-distr=
ibutedcache-is-deprecated-what-is-the-preferred-api">http://stackoverflow.c=
om/questions/21239722/hadoop-distributedcache-is-deprecated-what-is-the-pre=
ferred-api</a><br><div><br></div><div>Basically you can use the new method =
job.addCacheFiles to pass on stuff to the individual tasks.</div></div><div=
><br></div><div>Regards,</div><div>Shahab</div></div><div class=3D"gmail_ex=
tra"><br><div class=3D"gmail_quote">On Thu, Dec 11, 2014 at 9:07 PM, Sriniv=
as Chamarthi <span dir=3D"ltr">&lt;<a href=3D"mailto:srinivas.chamarthi@gma=
il.com" target=3D"_blank">srinivas.chamarthi@gmail.com</a>&gt;</span> wrote=
:<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1=
px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi,<div><br></div><div>I w=
ant to cache map/reducer temporary output files so that I can compare two m=
ap results coming from two different nodes to verify the integrity check.=
=C2=A0</div><div><br></div><div>I am simulating this use case with speculat=
ive execution by rescheduling the first task as soon as it is started and r=
unning.=C2=A0</div><div><br></div><div>Now I want to compare output files c=
oming from speculative attempt and prior attempt so that I can calculate th=
e credit scoring of each node.=C2=A0</div><div><br></div><div>I want to use=
 DistributedCache to cache the local file system files in CommitPending sta=
ge from TaskImpl. But the DistributedCache is actually deprecated. is there=
 any other way I can do this ?=C2=A0</div><div><br></div><div>I think I can=
 use HDFS to save the temporary output files so that other nodes can see it=
 ? but is there any in-memory solution I can use ?=C2=A0</div><div><br></di=
v><div>any pointers are greatly appreciated.=C2=A0</div><div><br></div><div=
>thx &amp; rgds,</div><div>srinivas chamarthi</div></div>
</blockquote></div></div>

--001a11c3d510baf4c50509fd46f8--