Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:subject:mime-version:date:references:x-mailer;
	b=Ua8E0bZFCx4eCAle1p0zgCoVF7tnVKNVM2Fia8AHp+RaeNuqL7H9qU4uK64cbidM
Message-Id: <0DCAB8AF-A35D-4CC6-87B3-A01F2A82C321@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: mapreduce-user@hadoop.apache.org
In-Reply-To: <875057.75594.qm@web15903.mail.cnb.yahoo.com>
Content-Type: text/plain; charset=GB2312; format=flowed; delsp=yes
Content-Transfer-Encoding: quoted-printable
Subject: Re: where distributed cache start working
Mime-Version: 1.0 (Apple Message framework v936)
Date: Sun, 22 Aug 2010 18:38:02 -0700
References: <346926.55544.qm@web15904.mail.cnb.yahoo.com>
 <AANLkTimXhkqpRPc_dBRcqJWsNLGEAABrF2rkLh-GFOjL@mail.gmail.com>
 <875057.75594.qm@web15903.mail.cnb.yahoo.com>

Moving to mapreduce-user@, bcc common-dev@. Please use the project =20
specific lists.

DistributedCache.purgeCache isn't a public api. You shouldn't be =20
calling it from the task.

  A simple way of doing what you want is to change the mtime of the =20
cache files on HDFS.

Arun

On Aug 22, 2010, at 9:48 AM, Gang Luo wrote:

> Thanks Jeff.
>
> However, are you sure TaskRunner.run() is also used in the new API? =20=

> I use btrace
> to trace the function call but didn't find this function had been =20
> called
> anywhere.
>
>
> One more question about distributed cache. After I call
> DistributedCache.purgeCache, I think the local cached files should =20
> be deleted or
> invalidated. However ,When I run the same job with the purge =20
> operation at the
> end multiple times, I find the local files have never been deleted =20
> and the
> modification time is when the first job run. How can I ask my job to
> re-distributed the cache again anyway?
>
> Thanks,
> -Gang
>
>
>
>
> ----- =D4=AD=CA=BC=D3=CA=BC=FE ----
> =B7=A2=BC=FE=C8=CB=A3=BA Jeff Zhang <zjffdu@gmail.com>
> =CA=D5=BC=FE=C8=CB=A3=BA common-dev@hadoop.apache.org
> =B7=A2=CB=CD=C8=D5=C6=DA=A3=BA 2010/8/20 (=D6=DC=CE=E5) 11:22:49 =C9=CF=CE=
=E7
> =D6=F7   =CC=E2=A3=BA Re: where distributed cache start working
>
> Hi Gang,
>
> In the TaskRunner's run() method, hadoop will download the cache files
> which you set on the client side to local, then the forked child jvm
> can use these cache files locally.
>
>
>
> On Fri, Aug 20, 2010 at 8:08 AM, Gang Luo <lgpublic@yahoo.com.cn> =20
> wrote:
>> Hi all,
>> I go through the code, but couldn't find the place where =20
>> distributed cache
>> start
>> working. I want to know between DistriubtedCache.addCacheFile at =20
>> the master
>> node
>> and DistributedCache.getLocalCacheFiles at the client side, when =20
>> and where are
>> the files get distributed.
>>
>>
>> Thanks,
>> -Gang
>>
>>
>>
>>
>>
>
>
>
> --=20
> Best Regards
>
> Jeff Zhang
>
>
>
>