hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From d...@apache.org
Subject svn commit: r654080 - in /hadoop/core/trunk: ./ docs/ src/docs/src/documentation/content/xdocs/ src/java/org/apache/hadoop/filecache/ src/java/org/apache/hadoop/fs/ src/test/org/apache/hadoop/dfs/ src/test/org/apache/hadoop/mapred/
Date Wed, 07 May 2008 12:23:54 GMT
Author: ddas
Date: Wed May  7 05:23:52 2008
New Revision: 654080

URL: http://svn.apache.org/viewvc?rev=654080&view=rev
Log:
HADOOP-2019. Adds support for .tar, .tgz and .tar.gz files in DistributedCache. Contributed
by Amareshwari Sriramadasu.

Added:
    hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar   (with props)
    hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar.gz   (with props)
    hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tgz   (with props)
Modified:
    hadoop/core/trunk/CHANGES.txt
    hadoop/core/trunk/build.xml
    hadoop/core/trunk/docs/changes.html
    hadoop/core/trunk/docs/mapred_tutorial.html
    hadoop/core/trunk/docs/mapred_tutorial.pdf
    hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
    hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java
    hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java
    hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java
    hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java

Modified: hadoop/core/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/CHANGES.txt (original)
+++ hadoop/core/trunk/CHANGES.txt Wed May  7 05:23:52 2008
@@ -59,6 +59,9 @@
     "hdfs:" URIs now defaults to 8020, so that one may simply use URIs
     of the form "hdfs://example.com/dir/file".
 
+    HADOOP-2019. Adds support for .tar, .tgz and .tar.gz files in 
+    DistributedCache (Amareshwari Sriramadasu via ddas)
+
   IMPROVEMENTS
    
     HADOOP-2928. Remove deprecated FileSystem.getContentLength().

Modified: hadoop/core/trunk/build.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/build.xml?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/build.xml (original)
+++ hadoop/core/trunk/build.xml Wed May  7 05:23:52 2008
@@ -528,6 +528,9 @@
     <copy file="${test.src.dir}/org/apache/hadoop/mapred/test.txt" todir="${test.cache.data}"/>
     <copy file="${test.src.dir}/org/apache/hadoop/mapred/test.jar" todir="${test.cache.data}"/>
     <copy file="${test.src.dir}/org/apache/hadoop/mapred/test.zip" todir="${test.cache.data}"/>
+    <copy file="${test.src.dir}/org/apache/hadoop/mapred/test.tar" todir="${test.cache.data}"/>
+    <copy file="${test.src.dir}/org/apache/hadoop/mapred/test.tgz" todir="${test.cache.data}"/>
+    <copy file="${test.src.dir}/org/apache/hadoop/mapred/test.tar.gz" todir="${test.cache.data}"/>
     <copy file="${test.src.dir}/org/apache/hadoop/dfs/hadoop-14-dfs-dir.tgz" todir="${test.cache.data}"/>
     <copy file="${test.src.dir}/org/apache/hadoop/dfs/hadoop-dfs-dir.txt" todir="${test.cache.data}"/>
   </target>

Modified: hadoop/core/trunk/docs/changes.html
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/docs/changes.html?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/docs/changes.html (original)
+++ hadoop/core/trunk/docs/changes.html Wed May  7 05:23:52 2008
@@ -91,7 +91,7 @@
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._new_features_')">
 NEW FEATURES
-</a>&nbsp;&nbsp;&nbsp;(4)
+</a>&nbsp;&nbsp;&nbsp;(6)
     <ol id="trunk_(unreleased_changes)_._new_features_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3074">HADOOP-3074</a>.
Provides a UrlStreamHandler for DFS and other FS,
 relying on FileSystem<br />(taton)</li>
@@ -101,10 +101,16 @@
 Bialecki via omalley)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2857">HADOOP-2857</a>.
Allow libhdfs to set jvm options.<br />(Craig Macdonald
 via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3317">HADOOP-3317</a>.
Add default port for HDFS namenode.  The port in
+"hdfs:" URIs now defaults to 8020, so that one may simply use URIs
+of the form "hdfs://example.com/dir/file".
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2019">HADOOP-2019</a>.
Adds support for .tar, .tgz and .tar.gz files in
+DistributedCache<br />(Amareshwari Sriramadasu via ddas)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._improvements_')">
 IMPROVEMENTS
-</a>&nbsp;&nbsp;&nbsp;(10)
+</a>&nbsp;&nbsp;&nbsp;(11)
     <ol id="trunk_(unreleased_changes)_._improvements_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2928">HADOOP-2928</a>.
Remove deprecated FileSystem.getContentLength().<br />(Lohit Vjayarenu via rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3130">HADOOP-3130</a>.
Make the connect timeout smaller for getFile.<br />(Amar Ramesh Kamat via ddas)</li>
@@ -129,6 +135,9 @@
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3144">HADOOP-3144</a>.
Improve robustness of LineRecordReader by defining a maximum
 line length (mapred.linerecordreader.maxlength), thereby avoiding reading
 too far into the following split.<br />(Zheng Shao via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3334">HADOOP-3334</a>.
Move lease handling from FSNamesystem into a seperate class.
+(Tsz Wo (Nicholas), SZE via rangadi)
+</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._optimizations_')">
 OPTIMIZATIONS

Modified: hadoop/core/trunk/docs/mapred_tutorial.html
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/docs/mapred_tutorial.html?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/docs/mapred_tutorial.html (original)
+++ hadoop/core/trunk/docs/mapred_tutorial.html Wed May  7 05:23:52 2008
@@ -1872,7 +1872,8 @@
 <p>
 <span class="codefrag">DistributedCache</span> can be used to distribute simple,

           read-only data/text files and more complex types such as archives and
-          jars. Archives (zip files) are <em>un-archived</em> at the slave nodes.
+          jars. Archives (zip, tar, tgz and tar.gz files) are 
+          <em>un-archived</em> at the slave nodes.
           Optionally users can also direct the <span class="codefrag">DistributedCache</span>
to 
           <em>symlink</em> the cached file(s) into the <span class="codefrag">current
working 
           directory</span> of the task via the 

Modified: hadoop/core/trunk/docs/mapred_tutorial.pdf
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/docs/mapred_tutorial.pdf?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/docs/mapred_tutorial.pdf (original)
+++ hadoop/core/trunk/docs/mapred_tutorial.pdf Wed May  7 05:23:52 2008
@@ -1519,10 +1519,10 @@
 >>
 endobj
 175 0 obj
-<< /Length 2600 /Filter [ /ASCII85Decode /FlateDecode ]
+<< /Length 2590 /Filter [ /ASCII85Decode /FlateDecode ]
  >>
 stream
-Gatm=D3N__&cR6oJfcbso1XhtqhqAsN8sQ#Au99WlW]Sa&-YYp;'ZO4B4%/-pFZ`/>`\rfefBh?6l^EU]6JMn=m5bMh4M0#s1%Zj'E!?(Qat%Wi])[b@3e'UU*(U^(cm7/qn'W#IK0<`n:MF(9s6cDArc&lde7qKp/3O@o,K%oq':,NB7"WI^&NXBXl0)ZAV[`O5I4G:DpJ&(:_8TnY/7oB&mq!fbq%GZ\)EoR+-/8A"`IbFQ!./5r4ifkar^MhQ2=l0r^uM$HZnUn[!!/U_ELuZ$-?/qk?M.R^<I&:%$'PC#8miO[dWWgp::,;3?[lKIqB-nqFCe`YXJ1DB`G*f]pbNb`iPc[\h85&>@*R(0q.;lOKU"F6#WNhCR4Re/UW3BVD.Dj/7ciCB9(9@'*V:=]0@Btc0L,ijKP"8TA-f/5N2t<Eb=*WWGI-b0_1$l`Oaeg<Vs7*H!Go0jE`)CMih`&mHUPucu)8,s#gao1i8]LU':KhgY7`YSL(b:`$4dKd85Bndm>a7d#sDHXDBUd9ot8gO/,iaerS!q',&?Z0(K9#I<]I0G$,-@QXi`\@g5>X3u+.MH+aQ4i^JD`?.HBj-s=iaL5M=e1.;jn$Mo]=<<n8"l<%bU:gsfGo7V9Hij@+ApfQorLMDW<PYB&SI:g#XX+%Khr(2N<DZ$>"-F)TnkF@etZ<EhF[TZWH)jk?rg3-ah]a!5k/7,\.jLSR+3>u<cAU_mW/XEnWCF7I-X@96JP\Xns\3'<[CiQC(2hQYRGin3.H\&S07=O_@j[t_S"jNYXM`I-h!;/nU*:sSnqUX(fNPqil7E)pIoS8J-ppOR0'_=^M*3ku-V8D$I`uptSY!?3W'ggo/iSI9k"LVucirf"j6qIdQ$K.Xd(53Y\0lp77bfeHO;%TVh=&:hM3iAf76sYG_K<sPkDMTL_06P)BAWf2iUZ_9#[e)kK/UKe!BmQc&"Y`WEW;u(Gh"NT[oVL+Md:rEU'q]b
 ,CV*6^LVWaunb2DMT=gb@K#u><^)nEV-/iiL&uD;iN)h[7LD\#PM;hJN[?fUACDju+qbuU@]`GbB[p%csa#!btZX=5%ZtM:H#aWdGZ6J#K,*GNIYX/9mW'`3`hHXCHeK0*&-5!ni8Mi)eJsT96JWF67R@(8?)58-`;rs9Hg"eSJ<F5/)48Xn@P%DA11[fun,VO=@#m,\][:EH;Vc$+O@s'L3K;E<uKpTHl6<GTGqN4clGM^)UeC5B@@:+U=:>c5.&)qT*E*SPqM=\jbTOuhB<1KI]7"]/f$Zih>*9p<-FL49]h7]='po+pT)4,@39uA2qi.lq*81gr(8r?-P=IlA.14_!tP('4S'FlE1[h9f8(Z$u`[Tf/A@M4#DD%19Y<fpKfKbR3Z,nQ8n"F*,1WWF9+G-_`jJ,qmJhCWuG:K$Rt@;2R$9-]7gO,1C[#G)/*+2chV$/t90irK2_f^HCl$PGoTGg"2XI)''M%-l6POEO:tIDlArX6WBb2I^c_/lAR[1(^)l/",>k[.#;'<#'-]d'#'p@07u5pT:im!,LI&@MP?BT>jG1&hDKY\BQJ\FQJ&\9\(o8d=rOf]hI1M@Nn<u4jE0\<J#@)Em%&naK'on7gE)!'5_ScC]1K'?E]r\:54^p%\P*Q?m3256TRoWf#/Y3DLqF[#P.GV6Bu<.bqHk+jIihEGR0/$G!iQ`<ON_'+G`UhdP$5>&]NeQc%S-=Q];oIn%05!-))o6rU[M,$8C43g5rGkYZ-b0<;7Un7j(i5'k;]OU<*6Ccpc++YJ1rZKJ/g)Y8TuGUf$<e4o,_M3%c\+f<fdqVW-*2TRiT9!s;>EZb-_B\r*91e^n.Z5Ai0?c;!0Qc!B#A+*64HVfg+(^,*=\_p>SCD14a=C/I8qCg[269T.1?S0Scd2oc+o(KF4ghb+<m"B,,#eoJ)(W/TacDEbK:'elQY,;Kq$&o4E<TmpY#U;p"JinKcr.f.5bjKLOJhZq1%Psa
 "(N;E/3s*4tK?k)`naIZLV\7&fsI,$#aam=?T8I$?]2JYH5EJGP^.Eco9^j>AA/8&!+`#dRF6RCL:Du_(05qhA2nZq$OTcRnMbfph@aH#>t=Htld`\BPq(XctR5'#&Dkm1DH(T&E[VQd3N`bKOiOBDh:h^oK=<JPjA$%#c_/SIO->f\9H,0aA+7CP117;b_LRE095$k7uVA0d,;4Eg\/)>*d!-IG<`\P?nnb$17@6\/S1c.))dR/n"u`#i=]=oWO+<1+?"?SZCu_jK$!mL*[:W:E2je!8;"D;9569sR5Ga?5VW6m,jGYiAcL]JV5d&BL:#+&LD[e^GAd0I#p-J;-pr;a;CAcIFHgJX^E>DpQtLobPt_CkQ-QNV2_/WH%!SGnZU;!MJWE*ni'[&)[:X54QP*+4PKP_j8$#P&R[9!Ys;X1UZ>oih6fn8aOc]jLDki+NBoe3WhIC`Z`+98YVp/\548XV`A(1KkIb;$V?cqB)FtkqJKVo]^X^Mdu`'T_?b@F87)'Z%a\2m;O">n<W+-:l.jn07"1&B^[i[gHLb?#R@XTiU`SCBf1TQFr1Q0[T8'Ck[IRFB_">YjA.V[H)6&C?mfkp2i7Q@P?`2!D#LB`0XT~>
+Gatm=D3N__&cR6oJfcbso1Xf^_08\E)9;!PbtQU9f!d%L+;0jqj'+h?lq>h-hBbIE$OWQW]&U&D`,QU*Mqr$<"i&sOrHWbacTBH5_V]Q]WCpJDMB_gS#`#$2Tc]IT#TUsWrS9^+^An3@:d9O2k1:[r;obZ;Zp!3TUQc.94@_ttPXKEICeNrXq]A5\8[7_GT"ejiM8+([FgaMdK(,Mm/54lQ:gI83Y@-tLL3kiYG[Qpf7/&I?1pVQG3R?lR4o0"Sm!F3V)f%A#H/<1Im3#QR5Lo<2ajfq.RBlb][O?A^3MLlgV^+7V^:Wk/kO)R8JBUdJRC?ko,Epct$JF=2rTe4mC4<0B[sH2cV@XX26s[4=BVG_t5+D<O]o@2QGU<p/n4h2t_R/KaO<Gj@"ZjD)(.cF4rfeMYf+/D5@6+@8lFI1KM'F.CKlL$dV,oYlKH?h$Uq?S8]mj6fLQtgI?&qBH0f![Y?/8H`g'ro9^K1H%Ao6:h>0p'H]Bsjm=UOId>X_s`)CCS]<_1MVXlt#QWt+<l.HSFr,K\W*p#GVuQpafEM`cSa+(!Z]QRcM9iq-F&=92jsc2b16XeWZZ;%tFXlc)s@(!h-Z1APrhV.cT?_4Ip/3p4#M%^gHc't/Y#:M>Lc/fSYcO7dsAW.qg8nI/b%1XCele^)g:ItPS92ZDDRPPF)Z1;U""=Zr9#[T_7Y/X]bmflgZEH7a""Wd)pO;!;e.k%Fn6>knhq[TI"[ekBQM7>k>`.-Jm+o'Rd/PJs60g.VS!j&p!GV&*7m.]!&F^m-8&8X;NI(3@VZ#<YmI#JD;]h`=dr8?>!W<%M7sGk`Zfh\Ma(@TfXa,BhNR.a0LD'#[`)(MM!l7[^d$Epk=rJm,RmnH"!pZ=3u\"nJDG(55q$@cjJD=]EEpAjMM(?U<L)l&8X)&G7Ub&$CffZ6>+<.%>f5au&a`NF('ole'*?[d8NpbI"@q!#WZ5%"MJDm/%Y<qU0$pd;/ESJ^:W
 h[G;)1LQ_>Te_@_5haE):m[1pga5:^s_^*(5*(,6Zdu=/7O-/Q%,<DTkL,8PC$d-pDcpb'%D<Q+%)Ll[[EmR`+ncb4f/[&:(CAB*"[bTP2%6Yef+PlJc,3;h;Je[0M]b4YtV8W*Q?(u`r$W0'h75s^,RP)OmLk`P=<i=,Bgqt$F$=dEM4A+?^l_RQK06.d9!D*>TR28idl[f%;I2Q!jWOAQ!^hkWuFnT2_>EsBprB!MDY(7[R0N:(s"WBT.I9J'>QR$#+Hu_EL3Hc%#7qZhEIQ4fkAQ"Tu0W5(G3Zd/GI[DG"Z'Q]"%ri[c_dfrIK?4V!Q8J_TrhBs*Vm/psg#T($37_aM-^_sA"Gf]"g/cIB/(76],&a0Wjk\]E@S?P.Jk;[>+qbP2cU`1ai+(2Q(]4tfJ0sIV"YZModI%f%"m76V7$8bbDTi4/K7f6Cm=5?/G3O75!C#jo?F_?q/kYW!ZW$XL4[5OfUN`Eu\-*.siNI0g>';fEF94]\<*^A4d#hC7MJQVo$d7kI_;)IM6,Nn+@/eno[fP(:mmeia90\E?"GN-)'#NcFn@C)^&jm@F3SH`.\sqqR;bR<#0rX2M^.P]VD!m_npqq6jfqY&Rr5^o<h-*%ECTgf87jg^ESG%N*pg>8X25YJWJ?:Q>XPAueB-#(:G:KXmkA>\QFtsIf<uE&EiMbRom"if-+GP>8&ljB0!(@`1<c/+NJ%0aA8bp*(^g85^C,\>+CI<WkNe7f*\dd7@dY/#H.'@&[EN,"W-o1f)FHPk`bHJo$5m)U7N?g?0VfU]LqbS=?$n>E;G'Krto%9;U^s,r4^8JWS^(C*$`6jt8lS$,9ld*qgai=4g1V<ElL0O#W5&d(]9J03;pOZ).+-<?hf$G,CQTmX>U:1EKnXSU.6]7au3b-.t'n]lqF+$2Z\QK(Yj5gt6+1j$<0WU1C17+FBCT"#U^J\amQVJLX)jduc`lP+<YVocZ_(j$RWTca0dcDj,AI2
 ;>#'qCk:S^&,ZuD<+\4bJ3R0)DD-6LGiCHXOuV]BD!bP)!E4ncLR'/*2%k'Z&2a>Pmq8<KZXV7B^@FS((",Ag&`Q!t87-V^S;arCoOW'/@/$Y_F>*e\@W#Qu\rNU3S%#ZFjW'B]fb0W;(CN;&55J*o#]H:W4jd"5jQ_rMP&,[@k+aaS*\$JgC1$^'7%rYZ1p:.Z"fH0kB]1[(Zd-8@si\P='o%6$"WLB>0A>H$5g1+,nc&3@MqPQ$-s=IV*K^M)3$LG>o6g_h_^;MP@^9ik5^WH$Bi29?b:5D?(Za9^km(3\I_aBQia(]q70qu\GF&Nu+^%79&L!b`E.3X+U'k#tK-ik(&Erq9Og?11D^MiH*T]aN\F1H1=F?T+&B)%uT6h*X+kpu.]l%8S2JpOi=60l9LQKEP@<`dk97:">PNMha3X`XuL.jCNVk6];1!F9ZnfNFVaH'8KMeE<./9:3/Dh$I<BU)NZ59bkRSHp"d7pH/Z;4V]cA+K+N6d0I1?H3^?bT85ZtTf$0*ea4#j&naIk%IRcbDI\NtacVG2X;@d]7o>Rq[r_O]J:XaI!>'[2r@/?35A.hh5NRYkUGDi%QE9q5Qf/[mO!U3.0kl~>
 endstream
 endobj
 176 0 obj
@@ -2855,53 +2855,53 @@
 xref
 0 319
 0000000000 65535 f 
-0000114936 00000 n 
-0000115217 00000 n 
-0000115310 00000 n 
+0000114926 00000 n 
+0000115207 00000 n 
+0000115300 00000 n 
 0000000015 00000 n 
 0000000071 00000 n 
 0000001299 00000 n 
 0000001419 00000 n 
 0000001577 00000 n 
-0000115462 00000 n 
+0000115452 00000 n 
 0000001712 00000 n 
-0000115525 00000 n 
+0000115515 00000 n 
 0000001849 00000 n 
-0000115591 00000 n 
+0000115581 00000 n 
 0000001986 00000 n 
-0000115657 00000 n 
+0000115647 00000 n 
 0000002123 00000 n 
-0000115721 00000 n 
+0000115711 00000 n 
 0000002259 00000 n 
-0000115787 00000 n 
+0000115777 00000 n 
 0000002396 00000 n 
-0000115853 00000 n 
+0000115843 00000 n 
 0000002533 00000 n 
-0000115917 00000 n 
+0000115907 00000 n 
 0000002669 00000 n 
-0000115981 00000 n 
+0000115971 00000 n 
 0000002806 00000 n 
-0000116045 00000 n 
+0000116035 00000 n 
 0000002943 00000 n 
-0000116109 00000 n 
+0000116099 00000 n 
 0000003078 00000 n 
-0000116176 00000 n 
+0000116166 00000 n 
 0000003215 00000 n 
-0000116241 00000 n 
+0000116231 00000 n 
 0000003352 00000 n 
-0000116306 00000 n 
+0000116296 00000 n 
 0000003487 00000 n 
-0000116371 00000 n 
+0000116361 00000 n 
 0000003624 00000 n 
-0000116438 00000 n 
+0000116428 00000 n 
 0000003761 00000 n 
-0000116505 00000 n 
+0000116495 00000 n 
 0000003897 00000 n 
-0000116570 00000 n 
+0000116560 00000 n 
 0000004034 00000 n 
-0000116637 00000 n 
+0000116627 00000 n 
 0000004171 00000 n 
-0000116702 00000 n 
+0000116692 00000 n 
 0000004308 00000 n 
 0000006939 00000 n 
 0000007062 00000 n 
@@ -2994,7 +2994,7 @@
 0000051557 00000 n 
 0000051602 00000 n 
 0000051811 00000 n 
-0000116767 00000 n 
+0000116757 00000 n 
 0000051947 00000 n 
 0000052238 00000 n 
 0000054964 00000 n 
@@ -3030,149 +3030,149 @@
 0000067897 00000 n 
 0000068125 00000 n 
 0000068330 00000 n 
-0000071024 00000 n 
-0000071150 00000 n 
-0000071211 00000 n 
-0000071464 00000 n 
-0000071654 00000 n 
-0000071859 00000 n 
-0000072106 00000 n 
-0000072307 00000 n 
-0000074778 00000 n 
-0000074904 00000 n 
-0000074965 00000 n 
-0000075240 00000 n 
-0000075515 00000 n 
-0000075773 00000 n 
-0000076002 00000 n 
-0000076237 00000 n 
-0000078926 00000 n 
-0000079052 00000 n 
-0000079201 00000 n 
-0000079412 00000 n 
-0000079620 00000 n 
-0000079795 00000 n 
-0000079990 00000 n 
-0000080164 00000 n 
-0000080340 00000 n 
-0000080563 00000 n 
-0000080804 00000 n 
-0000081000 00000 n 
-0000081213 00000 n 
-0000081430 00000 n 
-0000081649 00000 n 
-0000081921 00000 n 
-0000082186 00000 n 
-0000082466 00000 n 
-0000082679 00000 n 
-0000085278 00000 n 
-0000085404 00000 n 
-0000085457 00000 n 
-0000085778 00000 n 
-0000086096 00000 n 
-0000086282 00000 n 
-0000086480 00000 n 
-0000088762 00000 n 
-0000088872 00000 n 
-0000091219 00000 n 
-0000091329 00000 n 
-0000093653 00000 n 
-0000093763 00000 n 
-0000096102 00000 n 
-0000096212 00000 n 
-0000098426 00000 n 
-0000098536 00000 n 
-0000100530 00000 n 
-0000100640 00000 n 
-0000101895 00000 n 
-0000102005 00000 n 
-0000103719 00000 n 
-0000116827 00000 n 
-0000103829 00000 n 
-0000103965 00000 n 
-0000104158 00000 n 
-0000104316 00000 n 
-0000104532 00000 n 
-0000104816 00000 n 
-0000104986 00000 n 
-0000105136 00000 n 
-0000105312 00000 n 
-0000105628 00000 n 
-0000116881 00000 n 
-0000105818 00000 n 
-0000116948 00000 n 
-0000106012 00000 n 
-0000117013 00000 n 
-0000106204 00000 n 
-0000117080 00000 n 
-0000106419 00000 n 
-0000117147 00000 n 
-0000106587 00000 n 
-0000117214 00000 n 
-0000106794 00000 n 
-0000117279 00000 n 
-0000106998 00000 n 
-0000117345 00000 n 
-0000107175 00000 n 
-0000117412 00000 n 
-0000107415 00000 n 
-0000117479 00000 n 
-0000107612 00000 n 
-0000117545 00000 n 
-0000107809 00000 n 
-0000117613 00000 n 
-0000107988 00000 n 
-0000108194 00000 n 
-0000108415 00000 n 
-0000108699 00000 n 
-0000117681 00000 n 
-0000109032 00000 n 
-0000109198 00000 n 
-0000117749 00000 n 
-0000109413 00000 n 
-0000117815 00000 n 
-0000109589 00000 n 
-0000109777 00000 n 
-0000117883 00000 n 
-0000109998 00000 n 
-0000117951 00000 n 
-0000110243 00000 n 
-0000110431 00000 n 
-0000118017 00000 n 
-0000110703 00000 n 
-0000118085 00000 n 
-0000110867 00000 n 
-0000118153 00000 n 
-0000111094 00000 n 
-0000118219 00000 n 
-0000111249 00000 n 
-0000118287 00000 n 
-0000111470 00000 n 
-0000118353 00000 n 
-0000111697 00000 n 
-0000118421 00000 n 
-0000111998 00000 n 
-0000118489 00000 n 
-0000112261 00000 n 
-0000118555 00000 n 
-0000112487 00000 n 
-0000118623 00000 n 
-0000112678 00000 n 
-0000118691 00000 n 
-0000112931 00000 n 
-0000118759 00000 n 
-0000113176 00000 n 
-0000113367 00000 n 
-0000113636 00000 n 
-0000113806 00000 n 
-0000113991 00000 n 
-0000114156 00000 n 
-0000114270 00000 n 
-0000114381 00000 n 
-0000114493 00000 n 
-0000114602 00000 n 
-0000114709 00000 n 
-0000114826 00000 n 
+0000071014 00000 n 
+0000071140 00000 n 
+0000071201 00000 n 
+0000071454 00000 n 
+0000071644 00000 n 
+0000071849 00000 n 
+0000072096 00000 n 
+0000072297 00000 n 
+0000074768 00000 n 
+0000074894 00000 n 
+0000074955 00000 n 
+0000075230 00000 n 
+0000075505 00000 n 
+0000075763 00000 n 
+0000075992 00000 n 
+0000076227 00000 n 
+0000078916 00000 n 
+0000079042 00000 n 
+0000079191 00000 n 
+0000079402 00000 n 
+0000079610 00000 n 
+0000079785 00000 n 
+0000079980 00000 n 
+0000080154 00000 n 
+0000080330 00000 n 
+0000080553 00000 n 
+0000080794 00000 n 
+0000080990 00000 n 
+0000081203 00000 n 
+0000081420 00000 n 
+0000081639 00000 n 
+0000081911 00000 n 
+0000082176 00000 n 
+0000082456 00000 n 
+0000082669 00000 n 
+0000085268 00000 n 
+0000085394 00000 n 
+0000085447 00000 n 
+0000085768 00000 n 
+0000086086 00000 n 
+0000086272 00000 n 
+0000086470 00000 n 
+0000088752 00000 n 
+0000088862 00000 n 
+0000091209 00000 n 
+0000091319 00000 n 
+0000093643 00000 n 
+0000093753 00000 n 
+0000096092 00000 n 
+0000096202 00000 n 
+0000098416 00000 n 
+0000098526 00000 n 
+0000100520 00000 n 
+0000100630 00000 n 
+0000101885 00000 n 
+0000101995 00000 n 
+0000103709 00000 n 
+0000116817 00000 n 
+0000103819 00000 n 
+0000103955 00000 n 
+0000104148 00000 n 
+0000104306 00000 n 
+0000104522 00000 n 
+0000104806 00000 n 
+0000104976 00000 n 
+0000105126 00000 n 
+0000105302 00000 n 
+0000105618 00000 n 
+0000116871 00000 n 
+0000105808 00000 n 
+0000116938 00000 n 
+0000106002 00000 n 
+0000117003 00000 n 
+0000106194 00000 n 
+0000117070 00000 n 
+0000106409 00000 n 
+0000117137 00000 n 
+0000106577 00000 n 
+0000117204 00000 n 
+0000106784 00000 n 
+0000117269 00000 n 
+0000106988 00000 n 
+0000117335 00000 n 
+0000107165 00000 n 
+0000117402 00000 n 
+0000107405 00000 n 
+0000117469 00000 n 
+0000107602 00000 n 
+0000117535 00000 n 
+0000107799 00000 n 
+0000117603 00000 n 
+0000107978 00000 n 
+0000108184 00000 n 
+0000108405 00000 n 
+0000108689 00000 n 
+0000117671 00000 n 
+0000109022 00000 n 
+0000109188 00000 n 
+0000117739 00000 n 
+0000109403 00000 n 
+0000117805 00000 n 
+0000109579 00000 n 
+0000109767 00000 n 
+0000117873 00000 n 
+0000109988 00000 n 
+0000117941 00000 n 
+0000110233 00000 n 
+0000110421 00000 n 
+0000118007 00000 n 
+0000110693 00000 n 
+0000118075 00000 n 
+0000110857 00000 n 
+0000118143 00000 n 
+0000111084 00000 n 
+0000118209 00000 n 
+0000111239 00000 n 
+0000118277 00000 n 
+0000111460 00000 n 
+0000118343 00000 n 
+0000111687 00000 n 
+0000118411 00000 n 
+0000111988 00000 n 
+0000118479 00000 n 
+0000112251 00000 n 
+0000118545 00000 n 
+0000112477 00000 n 
+0000118613 00000 n 
+0000112668 00000 n 
+0000118681 00000 n 
+0000112921 00000 n 
+0000118749 00000 n 
+0000113166 00000 n 
+0000113357 00000 n 
+0000113626 00000 n 
+0000113796 00000 n 
+0000113981 00000 n 
+0000114146 00000 n 
+0000114260 00000 n 
+0000114371 00000 n 
+0000114483 00000 n 
+0000114592 00000 n 
+0000114699 00000 n 
+0000114816 00000 n 
 trailer
 <<
 /Size 319
@@ -3180,5 +3180,5 @@
 /Info 4 0 R
 >>
 startxref
-118827
+118817
 %%EOF

Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml Wed May
 7 05:23:52 2008
@@ -1416,7 +1416,8 @@
 
           <p><code>DistributedCache</code> can be used to distribute simple,

           read-only data/text files and more complex types such as archives and
-          jars. Archives (zip files) are <em>un-archived</em> at the slave nodes.
+          jars. Archives (zip, tar, tgz and tar.gz files) are 
+          <em>un-archived</em> at the slave nodes.
           Optionally users can also direct the <code>DistributedCache</code>
to 
           <em>symlink</em> the cached file(s) into the <code>current working

           directory</code> of the task via the 

Modified: hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java (original)
+++ hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java Wed May 
7 05:23:52 2008
@@ -50,11 +50,11 @@
  *
  * <p><code>DistributedCache</code> can be used to distribute simple, read-only
  * data/text files and/or more complex types such as archives, jars etc. 
- * Archives (zip files) are un-archived at the slave nodes. Jars maybe be 
- * optionally added to the classpath of the tasks, a rudimentary software
- * distribution mechanism.  Files have execution permissions. Optionally users 
- * can also direct it to symlink the distributed cache file(s) into 
- * the working directory of the task.</p>
+ * Archives (zip, tar and tgz/tar.gz files) are un-archived at the slave nodes. 
+ * Jars may be optionally added to the classpath of the tasks, a rudimentary 
+ * software distribution mechanism.  Files have execution permissions.
+ * Optionally users can also direct it to symlink the distributed cache file(s)
+ * into the working directory of the task.</p>
  * 
  * <p><code>DistributedCache</code> tracks modification timestamps of the
cache 
  * files. Clearly the cache files should not be modified by the application 
@@ -70,6 +70,9 @@
  *     $ bin/hadoop fs -copyFromLocal lookup.dat /myapp/lookup.dat  
  *     $ bin/hadoop fs -copyFromLocal map.zip /myapp/map.zip  
  *     $ bin/hadoop fs -copyFromLocal mylib.jar /myapp/mylib.jar
+ *     $ bin/hadoop fs -copyFromLocal mytar.tar /myapp/mytar.tar
+ *     $ bin/hadoop fs -copyFromLocal mytgz.tgz /myapp/mytgz.tgz
+ *     $ bin/hadoop fs -copyFromLocal mytargz.tar.gz /myapp/mytargz.tar.gz
  *     
  *     2. Setup the application's <code>JobConf</code>:
  *     
@@ -78,7 +81,10 @@
  *                                   job);
  *     DistributedCache.addCacheArchive(new URI("/myapp/map.zip", job);
  *     DistributedCache.addFileToClassPath(new Path("/myapp/mylib.jar"), job);
- *
+ *     DistributedCache.addCacheArchive(new URI("/myapp/mytar.tar", job);
+ *     DistributedCache.addCacheArchive(new URI("/myapp/mytgz.tgz", job);
+ *     DistributedCache.addCacheArchive(new URI("/myapp/mytargz.tar.gz", job);
+ *     
  *     3. Use the cached files in the {@link Mapper} or {@link Reducer}:
  *     
  *     public static class MapClass extends MapReduceBase  
@@ -129,9 +135,11 @@
    * @param conf The Confguration file which contains the filesystem
    * @param baseDir The base cache Dir where you wnat to localize the files/archives
    * @param fileStatus The file status on the dfs.
-   * @param isArchive if the cache is an archive or a file. In case it is an archive
-   *  with a .zip or .jar extension it will be unzipped/unjarred automatically 
-   *  and the directory where the archive is unjarred is returned as the Path.
+   * @param isArchive if the cache is an archive or a file. In case it is an
+   *  archive with a .zip or .jar or .tar or .tgz or .tar.gz extension it will
+   *  be unzipped/unjarred/untarred automatically 
+   *  and the directory where the archive is unzipped/unjarred/untarred is
+   *  returned as the Path.
    *  In case of a file, the path to the file is returned
    * @param confFileStamp this is the hdfs file modification timestamp to verify that the

    * file to be cached hasn't changed since the job started
@@ -185,9 +193,11 @@
    * being used in the Configuration
    * @param conf The Confguration file which contains the filesystem
    * @param baseDir The base cache Dir where you wnat to localize the files/archives
-   * @param isArchive if the cache is an archive or a file. In case it is an archive
-   *  with a .zip or .jar extension it will be unzipped/unjarred automatically 
-   *  and the directory where the archive is unjarred is returned as the Path.
+   * @param isArchive if the cache is an archive or a file. In case it is an 
+   *  archive with a .zip or .jar or .tar or .tgz or .tar.gz extension it will 
+   *  be unzipped/unjarred/untarred automatically 
+   *  and the directory where the archive is unzipped/unjarred/untarred 
+   *  is returned as the Path.
    *  In case of a file, the path to the file is returned
    * @param confFileStamp this is the hdfs file modification timestamp to verify that the

    * file to be cached hasn't changed since the job started
@@ -331,13 +341,14 @@
       fs.copyToLocalFile(new Path(cacheId), parchive);
       if (isArchive) {
         String tmpArchive = parchive.toString().toLowerCase();
+        File srcFile = new File(parchive.toString());
+        File destDir = new File(parchive.getParent().toString());
         if (tmpArchive.endsWith(".jar")) {
-          RunJar.unJar(new File(parchive.toString()), new File(parchive
-                                                               .getParent().toString()));
+          RunJar.unJar(srcFile, destDir);
         } else if (tmpArchive.endsWith(".zip")) {
-          FileUtil.unZip(new File(parchive.toString()), new File(parchive
-                                                                 .getParent().toString()));
-
+          FileUtil.unZip(srcFile, destDir);
+        } else if (isTarFile(tmpArchive)) {
+          FileUtil.unTar(srcFile, destDir);
         }
         // else will not do anyhting
         // and copy the file into the dir as it is
@@ -373,6 +384,11 @@
     }
   }
 
+  private static boolean isTarFile(String filename) {
+    return (filename.endsWith(".tgz") || filename.endsWith(".tar.gz") ||
+           filename.endsWith(".tar"));
+  }
+  
   // Checks if the cache has already been localized and is fresh
   private static boolean ifExistsAndFresh(Configuration conf, FileSystem fs, 
                                           URI cache, long confFileStamp, 

Modified: hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java (original)
+++ hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java Wed May  7 05:23:52 2008
@@ -26,6 +26,7 @@
 import org.apache.hadoop.io.IOUtils;
 import org.apache.hadoop.util.StringUtils;
 import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.util.Shell.ShellCommandExecutor;
 
 /**
  * A collection of file-processing util methods
@@ -453,7 +454,51 @@
       zipFile.close();
     }
   }
-  
+
+  /**
+   * Given a Tar File as input it will untar the file in a the untar directory
+   * passed as the second parameter
+   * 
+   * This utility will untar ".tar" files and ".tar.gz","tgz" files.
+   *  
+   * @param inFile The tar file as input. 
+   * @param untarDir The untar directory where to untar the tar file.
+   * @throws IOException
+   */
+  public static void unTar(File inFile, File untarDir) throws IOException {
+    if (!untarDir.mkdirs()) {           
+      if (!untarDir.isDirectory()) {
+        throw new IOException("Mkdirs failed to create " + untarDir);
+      }
+    }
+
+    StringBuffer untarCommand = new StringBuffer();
+    boolean gzipped = inFile.toString().endsWith("gz");
+    if (gzipped) {
+      untarCommand.append(" gzip -dc '");
+      untarCommand.append(FileUtil.makeShellPath(inFile));
+      untarCommand.append("' | (");
+    } 
+    untarCommand.append("cd '");
+    untarCommand.append(FileUtil.makeShellPath(untarDir)); 
+    untarCommand.append("' ; ");
+    untarCommand.append("tar -xf ");
+    
+    if (gzipped) {
+      untarCommand.append(" -)");
+    } else {
+      untarCommand.append(FileUtil.makeShellPath(inFile));
+    }
+    String[] shellCmd = { "bash", "-c", untarCommand.toString() };
+    ShellCommandExecutor shexec = new ShellCommandExecutor(shellCmd);
+    shexec.execute();
+    int exitcode = shexec.getExitCode();
+    if (exitcode != 0) {
+      throw new IOException("Error untarring file " + inFile + 
+                  ". Tar process exited with exit code " + exitcode);
+    }
+  }
+
   /**
    * Class for creating hardlinks.
    * Supports Unix, Cygwin, WindXP.

Modified: hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java (original)
+++ hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java Wed May
 7 05:23:52 2008
@@ -27,7 +27,6 @@
 import java.util.zip.CRC32;
 
 import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.util.Shell;
 import org.apache.hadoop.fs.FSInputStream;
 import org.apache.hadoop.fs.FileUtil;
 import org.apache.hadoop.dfs.FSConstants.StartupOption;
@@ -72,12 +71,7 @@
     if ( dfsDir.exists() && !FileUtil.fullyDelete(dfsDir) ) {
       throw new IOException("Could not delete dfs directory '" + dfsDir + "'");
     }
-    String cmd = "gzip -dc '" + FileUtil.makeShellPath(tarFile) + "' | (cd '" +
-                 FileUtil.makeShellPath(dataDir) + "' ; tar -xf -)";
-    LOG.info("Unpacking the tar file. Cmd : " + cmd);
-    String[] shellCmd = { "bash", "-c", cmd };
-    Shell.execCommand(shellCmd);
-    
+    FileUtil.unTar(new File(tarFile), new File(dataDir));
     //Now read the reference info
     
     BufferedReader reader = new BufferedReader( 

Modified: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java (original)
+++ hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java Wed May  7 05:23:52
2008
@@ -175,6 +175,9 @@
     Path txtPath = new Path(localPath, new Path("test.txt"));
     Path jarPath = new Path(localPath, new Path("test.jar"));
     Path zipPath = new Path(localPath, new Path("test.zip"));
+    Path tarPath = new Path(localPath, new Path("test.tgz"));
+    Path tarPath1 = new Path(localPath, new Path("test.tar.gz"));
+    Path tarPath2 = new Path(localPath, new Path("test.tar"));
     Path cachePath = new Path(cacheDir);
     fs.delete(cachePath, true);
     if (!fs.mkdirs(cachePath)) {
@@ -183,13 +186,23 @@
     fs.copyFromLocalFile(txtPath, cachePath);
     fs.copyFromLocalFile(jarPath, cachePath);
     fs.copyFromLocalFile(zipPath, cachePath);
+    fs.copyFromLocalFile(tarPath, cachePath);
+    fs.copyFromLocalFile(tarPath1, cachePath);
+    fs.copyFromLocalFile(tarPath2, cachePath);
     // setting the cached archives to zip, jar and simple text files
     URI uri1 = fs.getUri().resolve(cachePath + "/test.jar");
     URI uri2 = fs.getUri().resolve(cachePath + "/test.zip");
     URI uri3 = fs.getUri().resolve(cachePath + "/test.txt");
+    URI uri4 = fs.getUri().resolve(cachePath + "/test.tgz");
+    URI uri5 = fs.getUri().resolve(cachePath + "/test.tar.gz");
+    URI uri6 = fs.getUri().resolve(cachePath + "/test.tar");
+
     DistributedCache.addCacheArchive(uri1, conf);
     DistributedCache.addCacheArchive(uri2, conf);
     DistributedCache.addCacheFile(uri3, conf);
+    DistributedCache.addCacheArchive(uri4, conf);
+    DistributedCache.addCacheArchive(uri5, conf);
+    DistributedCache.addCacheArchive(uri6, conf);
     RunningJob job = JobClient.runJob(conf);
     int count = 0;
     // after the job ran check to see if the the input from the localized cache
@@ -208,7 +221,7 @@
       }
       file.close();
     }
-    if (count != 3)
+    if (count != 6)
       return new TestResult(job, false);
 
     return new TestResult(job, true);

Added: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar?rev=654080&view=auto
==============================================================================
Binary file - no diff available.

Propchange: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar.gz
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar.gz?rev=654080&view=auto
==============================================================================
Binary file - no diff available.

Propchange: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar.gz
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tgz
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tgz?rev=654080&view=auto
==============================================================================
Binary file - no diff available.

Propchange: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tgz
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream



Mime
View raw message