Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=2JNRQi8jT5dOIvG0rrJ4vNeOu6ETTpY15oMoVAL9hvpJW0vUlwBsKsDae1vJRmbJn1Xe4cfoNtOoU1XBGZ9GHUwGAQcmQ7GIi4F8kjyLeuZwP8kkXrEroj7FQamKWCqOstYL30anorP6bMywxukESvhUB2atiEMvbaKgKItEaaI=;
Message-ID: <730425.37803.qm@web15903.mail.cnb.yahoo.com>
References: <346926.55544.qm@web15904.mail.cnb.yahoo.com>
 <AANLkTimXhkqpRPc_dBRcqJWsNLGEAABrF2rkLh-GFOjL@mail.gmail.com>
 <875057.75594.qm@web15903.mail.cnb.yahoo.com>
 <0DCAB8AF-A35D-4CC6-87B3-A01F2A82C321@yahoo-inc.com>
Date: Thu, 26 Aug 2010 12:59:10 +0800 (CST)
From: Gang Luo <lgpublic@yahoo.com.cn>
Subject: Re: where distributed cache start working
To: common-dev@hadoop.apache.org
In-Reply-To: <0DCAB8AF-A35D-4CC6-87B3-A01F2A82C321@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=gb2312
Content-Transfer-Encoding: quoted-printable

Thanks Arun. Change the mTime is a good idea. However, given a file (the pa=
th is =0A=0AA/B/C/D/file) distributed to all the nodes, if I just change th=
e mTime of file =0Ato a earlier time stamp, it will not be replaced next ti=
me. Should I also change =0Athe mTime for all the directories along the pat=
h (A, B, C and D). Whose =0Atimestamp is used by DistributedCache?=0A=0ATha=
nks.=0A-Gang=0A=0A=0A=0A=0A----- =D4=AD=CA=BC=D3=CA=BC=FE ----=0A=B7=A2=BC=
=FE=C8=CB=A3=BA Arun C Murthy <acm@yahoo-inc.com>=0A=CA=D5=BC=FE=C8=CB=A3=
=BA mapreduce-user@hadoop.apache.org=0A=B7=A2=CB=CD=C8=D5=C6=DA=A3=BA 2010/=
8/22 (=D6=DC=C8=D5) 9:38:02 =CF=C2=CE=E7=0A=D6=F7   =CC=E2=A3=BA Re: where =
distributed cache start working=0A=0AMoving to mapreduce-user@, bcc common-=
dev@. Please use the project specific =0Alists.=0A=0ADistributedCache.purge=
Cache isn't a public api. You shouldn't be calling it from =0A=0Athe task.=
=0A=0AA simple way of doing what you want is to change the mtime of the cac=
he files on =0A=0AHDFS.=0A=0AArun=0A=0AOn Aug 22, 2010, at 9:48 AM, Gang Lu=
o wrote:=0A=0A> Thanks Jeff.=0A> =0A> However, are you sure TaskRunner.run(=
) is also used in the new API? I use =0A>btrace=0A> to trace the function c=
all but didn't find this function had been called=0A> anywhere.=0A> =0A> =
=0A> One more question about distributed cache. After I call=0A> Distribute=
dCache.purgeCache, I think the local cached files should be deleted =0A>or=
=0A> invalidated. However ,When I run the same job with the purge operation=
 at the=0A> end multiple times, I find the local files have never been dele=
ted and the=0A> modification time is when the first job run. How can I ask =
my job to=0A> re-distributed the cache again anyway?=0A> =0A> Thanks,=0A> -=
Gang=0A> =0A> =0A> =0A> =0A> ----- =D4=AD=CA=BC=D3=CA=BC=FE ----=0A> =B7=A2=
=BC=FE=C8=CB=A3=BA Jeff Zhang <zjffdu@gmail.com>=0A> =CA=D5=BC=FE=C8=CB=A3=
=BA common-dev@hadoop.apache.org=0A> =B7=A2=CB=CD=C8=D5=C6=DA=A3=BA 2010/8/=
20 (=D6=DC=CE=E5) 11:22:49 =C9=CF=CE=E7=0A> =D6=F7   =CC=E2=A3=BA Re: where=
 distributed cache start working=0A> =0A> Hi Gang,=0A> =0A> In the TaskRunn=
er's run() method, hadoop will download the cache files=0A> which you set o=
n the client side to local, then the forked child jvm=0A> can use these cac=
he files locally.=0A> =0A> =0A> =0A> On Fri, Aug 20, 2010 at 8:08 AM, Gang =
Luo <lgpublic@yahoo.com.cn> wrote:=0A>> Hi all,=0A>> I go through the code,=
 but couldn't find the place where distributed cache=0A>> start=0A>> workin=
g. I want to know between DistriubtedCache.addCacheFile at the master=0A>> =
node=0A>> and DistributedCache.getLocalCacheFiles at the client side, when =
and where =0Aare=0A>> the files get distributed.=0A>> =0A>> =0A>> Thanks,=
=0A>> -Gang=0A>> =0A>> =0A>> =0A>> =0A>> =0A> =0A> =0A> =0A> --Best Regards=
=0A> =0A> Jeff Zhang=0A> =0A> =0A> =0A>=0A=0A=0A=0A