Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=ZxJgoXMkYd/478iqbE5dek7je3YFHfYk6NBs6JKLj5uaBc73COTOFuRrkfe+pkIpsMhavsA5yH26WAXT8ODeSEWyZmTC9VBNNPZB7mqizRpXkVYDPTC3fCH2wxHTCm7qlbZpcDndYuvpO15lBSisLDNbZJ4Fe/eYaR4h36LZp+A=;
Message-ID: <757890.14505.qm@web15903.mail.cnb.yahoo.com>
References: <C7C68108.1846C%rphulari@yahoo-inc.com>
Date: Thu, 18 Mar 2010 04:58:02 +0800 (CST)
From: Gang Luo <lgpublic@yahoo.com.cn>
Subject: Re: when to sent distributed cache file
To: common-user@hadoop.apache.org
In-Reply-To: <C7C68108.1846C%rphulari@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=gb2312
Content-Transfer-Encoding: quoted-printable

Thanks Ravi.=0A=0AHere are some observations. I run job1 to generate some d=
ata used by the following job2 without replication. The total size of the j=
ob 1 output is 25mb and is in 50 files. I use distributed cache to sent all=
 the files to nodes running job2 tasks. When job2 starts, it stayed at "map=
 0% reduce 0%" for 10 minutes. When the job1 output is in 10 files (using 1=
0 reducers in job1), the time consumed here are 2 minutes. =0A=0ASo, I thin=
k the time to distribute cache files is actually counted as part of the tot=
al time of the MR job. And in order to sent a cache file from HDFS to local=
 disk, it sent at least one block (64mb by default) even that file is only =
1mb. Is that right? If so, how much space that cache file takes on the loca=
l disk, 64mb or 1mb? =0A=0A-Gang=0A=0A=0A=0A----- =D4=AD=CA=BC=D3=CA=BC=FE =
----=0A=B7=A2=BC=FE=C8=CB=A3=BA Ravi Phulari <rphulari@yahoo-inc.com>=0A=CA=
=D5=BC=FE=C8=CB=A3=BA "common-user@hadoop.apache.org" <common-user@hadoop.a=
pache.org>; Gang Luo <lgpublic@yahoo.com.cn>=0A=B7=A2=CB=CD=C8=D5=C6=DA=A3=
=BA 2010/3/17 (=D6=DC=C8=FD) 3:52:24 =CF=C2=CE=E7=0A=D6=F7   =CC=E2=A3=BA R=
e: when to sent distributed cache file=0A=0AHello Gang,=0A      The framewo=
rk will copy the necessary files to the slave node  before any tasks for th=
e job are executed on that node.=0ANot sure if  time required to distribute=
 cache is counted in map reduce job time but it is included in job submissi=
on process in JobClient .=0A--=0ARavi=0A=0AOn 3/17/10 11:32 AM, "Gang Luo" =
<lgpublic@yahoo.com.cn> wrote:=0A=0AHi all,=0AI doubt when does hadoop dist=
ributes the cache files. The moment we call DistributedCache.addCacheFile()=
 ? Will the time to distribute caches be counted as part of the mapreduce j=
ob time?=0A=0AThanks,=0A-Gang=0A=0A=0A