Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A0097DB9D for ; Wed, 28 Nov 2012 11:13:10 +0000 (UTC) Received: (qmail 47273 invoked by uid 500); 28 Nov 2012 11:13:05 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 47192 invoked by uid 500); 28 Nov 2012 11:13:05 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 47172 invoked by uid 99); 28 Nov 2012 11:13:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Nov 2012 11:13:04 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,MIME_BASE64_TEXT,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [85.115.52.190] (HELO cluster-a.mailcontrol.com) (85.115.52.190) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Nov 2012 11:12:54 +0000 Received: from sportingindex.com (83-244-202-21.cust-83.exponential-e.net [83.244.202.21]) by rly06a.srv.mailcontrol.com (MailControl) with ESMTP id qASBCVHo025561 for ; Wed, 28 Nov 2012 11:12:31 GMT Received: from dss-protector.sig.ads (unknown [127.0.0.1]) by dss-protector.sig.ads (Service) with ESMTP id A62A4128013 for ; Wed, 28 Nov 2012 11:12:31 +0000 (GMT) Received: from GBGH-SVEXCHFE02.sig.ads (unknown [10.10.14.23]) by dss-protector.sig.ads (Service) with ESMTP id DA3A2128002 for ; Wed, 28 Nov 2012 11:12:30 +0000 (GMT) Received: from GBGH-EXCH-CMS.sig.ads ([fe80::dcac:17fe:e957:d280]) by GBGH-SVEXCHFE02.sig.ads ([fe80::69ae:bd72:c665:191c%10]) with mapi; Wed, 28 Nov 2012 11:12:30 +0000 From: Tony Burton To: "" Date: Wed, 28 Nov 2012 11:12:30 +0000 Subject: Map output compression in Hadoop 1.0.3 Thread-Topic: Map output compression in Hadoop 1.0.3 Thread-Index: Ac3NWEm6sjupBU9UQUKQhb/x1uPncA== Message-ID: <556325346CA26341B6F0530E07F90D96016C64CD967F@GBGH-EXCH-CMS.sig.ads> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: multipart/alternative; boundary="_000_556325346CA26341B6F0530E07F90D96016C64CD967FGBGHEXCHCMS_" MIME-Version: 1.0 X-Scanned-By: MailControl 11783.69 (www.mailcontrol.com) on 10.65.0.116 X-Virus-Checked: Checked by ClamAV on apache.org --_000_556325346CA26341B6F0530E07F90D96016C64CD967FGBGHEXCHCMS_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, Quick question: What's the best way to turn on Map Output Compression in Ha= doop 1.0.3? The tutorial at http://hadoop.apache.org/docs/r1.0.3/mapred_tut= orial.html says to use JobConf.setCompressMapOutput(boolean), but I'm using= o.a.h.mapreduce.Job rather than o.a.h.mapred.JobConf. Is it simply a case of using getConf.set("mapred.output.compress", true) th= en constructing my Job from the Configuration object, or is there more dire= ct way that I've missed? Thanks, Tony ********************************************************************** Please consider the environment before printing this email or attachments This email and any attachments are confidential, protected by copyright and= may be legally privileged. If you are not the intended recipient, then th= e dissemination or copying of this email is prohibited. If you have receive= d this in error, please notify the sender by replying by email and then del= ete the email completely from your system. Neither Sporting Index nor the = sender accepts responsibility for any virus, or any other defect which migh= t affect any computer or IT system into which the email is received and/or = opened. It is the responsibility of the recipient to scan the email and no= responsibility is accepted for any loss or damage arising in any way from = receipt or use of this email. Sporting Index Ltd is a company registered i= n England and Wales with company number 2636842, whose registered office is= at Gateway House, Milverton Street, London, SE11 4AP. Sporting Index Ltd = is authorised and regulated by the UK Financial Services Authority (reg. no= . 150404) and Gambling Commission (reg. no. 000-027343-R-308898-001). Any = financial promotion contained herein has been issued and approved by Sporting Index Ltd. Outbound email has been scanned for viruses and SPAM --_000_556325346CA26341B6F0530E07F90D96016C64CD967FGBGHEXCHCMS_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwi IHhtbG5zOm89InVybjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9m ZmljZSIgeG1sbnM6dz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZp Y2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNv bS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5v cmcvVFIvUkVDLWh0bWw0MCI+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj1Db250 ZW50LVR5cGUgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXVzLWFzY2lp Ij48bWV0YSBuYW1lPUdlbmVyYXRvciBjb250ZW50PSJNaWNyb3NvZnQgV29y ZCAxNCAoZmlsdGVyZWQgbWVkaXVtKSI+PHN0eWxlPjwhLS0KLyogRm9udCBE ZWZpbml0aW9ucyAqLwpAZm9udC1mYWNlCgl7Zm9udC1mYW1pbHk6Q2FsaWJy aTsKCXBhbm9zZS0xOjIgMTUgNSAyIDIgMiA0IDMgMiA0O30KQGZvbnQtZmFj ZQoJe2ZvbnQtZmFtaWx5OlRhaG9tYTsKCXBhbm9zZS0xOjIgMTEgNiA0IDMg NSA0IDQgMiA0O30KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8KcC5Nc29Ob3Jt YWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbAoJe21hcmdpbjowY207 CgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7Cglmb250LXNpemU6MTIuMHB0OwoJ Zm9udC1mYW1pbHk6IlRpbWVzIE5ldyBSb21hbiIsInNlcmlmIjt9CmE6bGlu aywgc3Bhbi5Nc29IeXBlcmxpbmsKCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7 Cgljb2xvcjpibHVlOwoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9CmE6 dmlzaXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZAoJe21zby1zdHls ZS1wcmlvcml0eTo5OTsKCWNvbG9yOnB1cnBsZTsKCXRleHQtZGVjb3JhdGlv bjp1bmRlcmxpbmU7fQpwCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5OwoJbXNv LW1hcmdpbi10b3AtYWx0OmF1dG87CgltYXJnaW4tcmlnaHQ6MGNtOwoJbXNv LW1hcmdpbi1ib3R0b20tYWx0OmF1dG87CgltYXJnaW4tbGVmdDowY207Cglm b250LXNpemU6MTIuMHB0OwoJZm9udC1mYW1pbHk6IlRpbWVzIE5ldyBSb21h biIsInNlcmlmIjt9CnAuTXNvTm9TcGFjaW5nLCBsaS5Nc29Ob1NwYWNpbmcs IGRpdi5Nc29Ob1NwYWNpbmcKCXttc28tc3R5bGUtcHJpb3JpdHk6MTsKCW1h cmdpbjowY207CgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7Cglmb250LXNpemU6 MTIuMHB0OwoJZm9udC1mYW1pbHk6IlRpbWVzIE5ldyBSb21hbiIsInNlcmlm Ijt9CnNwYW4uRW1haWxTdHlsZTE5Cgl7bXNvLXN0eWxlLXR5cGU6cGVyc29u YWwtcmVwbHk7Cglmb250LWZhbWlseToiVGFob21hIiwic2Fucy1zZXJpZiI7 Cgljb2xvcjpibGFjazsKCWZvbnQtd2VpZ2h0Om5vcm1hbDsKCWZvbnQtc3R5 bGU6bm9ybWFsOwoJdGV4dC1kZWNvcmF0aW9uOm5vbmUgbm9uZTt9Ci5Nc29D aHBEZWZhdWx0Cgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7Cglmb250 LWZhbWlseToiQ2FsaWJyaSIsInNhbnMtc2VyaWYiOwoJbXNvLWZhcmVhc3Qt bGFuZ3VhZ2U6RU4tVVM7fQpAcGFnZSBXb3JkU2VjdGlvbjEKCXtzaXplOjYx Mi4wcHQgNzkyLjBwdDsKCW1hcmdpbjo3Mi4wcHQgNzIuMHB0IDcyLjBwdCA3 Mi4wcHQ7fQpkaXYuV29yZFNlY3Rpb24xCgl7cGFnZTpXb3JkU2VjdGlvbjE7 fQotLT48L3N0eWxlPjwhLS1baWYgZ3RlIG1zbyA5XT48eG1sPgo8bzpzaGFw ZWRlZmF1bHRzIHY6ZXh0PSJlZGl0IiBzcGlkbWF4PSIxMDI2IiAvPgo8L3ht bD48IVtlbmRpZl0tLT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4KPG86c2hh cGVsYXlvdXQgdjpleHQ9ImVkaXQiPgo8bzppZG1hcCB2OmV4dD0iZWRpdCIg ZGF0YT0iMSIgLz4KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0t PjwvaGVhZD48Ym9keSBsYW5nPUVOLUdCIGxpbms9Ymx1ZSB2bGluaz1wdXJw bGU+PGRpdiBjbGFzcz1Xb3JkU2VjdGlvbjE+PHAgY2xhc3M9TXNvTm9ybWFs PjxzcGFuIHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiJU YWhvbWEiLCJzYW5zLXNlcmlmIjtjb2xvcjpibGFjayc+SGksPG86cD48L286 cD48L3NwYW4+PC9wPjxwIGNsYXNzPU1zb05vcm1hbD48c3BhbiBzdHlsZT0n Zm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseToiVGFob21hIiwic2Fucy1z ZXJpZiI7Y29sb3I6YmxhY2snPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwv cD48cCBjbGFzcz1Nc29Ob3JtYWw+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZTox MS4wcHQ7Zm9udC1mYW1pbHk6IlRhaG9tYSIsInNhbnMtc2VyaWYiO2NvbG9y OmJsYWNrJz5RdWljayBxdWVzdGlvbjogV2hhdCYjODIxNztzIHRoZSBiZXN0 IHdheSB0byB0dXJuIG9uIE1hcCBPdXRwdXQgQ29tcHJlc3Npb24gaW4gSGFk b29wIDEuMC4zPyBUaGUgdHV0b3JpYWwgYXQgPGEgaHJlZj0iaHR0cDovL2hh ZG9vcC5hcGFjaGUub3JnL2RvY3MvcjEuMC4zL21hcHJlZF90dXRvcmlhbC5o dG1sIj5odHRwOi8vaGFkb29wLmFwYWNoZS5vcmcvZG9jcy9yMS4wLjMvbWFw cmVkX3R1dG9yaWFsLmh0bWw8L2E+IHNheXMgdG8gdXNlIEpvYkNvbmYuc2V0 Q29tcHJlc3NNYXBPdXRwdXQoYm9vbGVhbiksIGJ1dCBJJiM4MjE3O20gdXNp bmcgby5hLmgubWFwcmVkdWNlLkpvYiByYXRoZXIgdGhhbiBvLmEuaC5tYXBy ZWQuSm9iQ29uZi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+PHAgY2xhc3M9TXNv Tm9ybWFsPjxzcGFuIHN0eWxlPSdmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFt aWx5OiJUYWhvbWEiLCJzYW5zLXNlcmlmIjtjb2xvcjpibGFjayc+PG86cD4m bmJzcDs8L286cD48L3NwYW4+PC9wPjxwIGNsYXNzPU1zb05vcm1hbD48c3Bh biBzdHlsZT0nZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseToiVGFob21h Iiwic2Fucy1zZXJpZiI7Y29sb3I6YmxhY2snPklzIGl0IHNpbXBseSBhIGNh c2Ugb2YgdXNpbmcgZ2V0Q29uZi5zZXQoJnF1b3Q7bWFwcmVkLm91dHB1dC5j b21wcmVzcyZxdW90OywgdHJ1ZSkgdGhlbiBjb25zdHJ1Y3RpbmcgbXkgSm9i IGZyb20gdGhlIENvbmZpZ3VyYXRpb24gb2JqZWN0LCBvciBpcyB0aGVyZSBt b3JlIGRpcmVjdCB3YXkgdGhhdCBJJiM4MjE3O3ZlIG1pc3NlZD8gPG86cD48 L286cD48L3NwYW4+PC9wPjxwIGNsYXNzPU1zb05vcm1hbD48c3BhbiBzdHls ZT0nZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseToiVGFob21hIiwic2Fu cy1zZXJpZiI7Y29sb3I6YmxhY2snPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFu PjwvcD48cCBjbGFzcz1Nc29Ob3JtYWw+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6 ZToxMS4wcHQ7Zm9udC1mYW1pbHk6IlRhaG9tYSIsInNhbnMtc2VyaWYiO2Nv bG9yOmJsYWNrJz5UaGFua3MsPG86cD48L286cD48L3NwYW4+PC9wPjxwIGNs YXNzPU1zb05vcm1hbD48c3BhbiBzdHlsZT0nZm9udC1zaXplOjExLjBwdDtm b250LWZhbWlseToiVGFob21hIiwic2Fucy1zZXJpZiI7Y29sb3I6YmxhY2sn PjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD48cCBjbGFzcz1Nc29Ob3Jt YWw+PHNwYW4gc3R5bGU9J2ZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6 IlRhaG9tYSIsInNhbnMtc2VyaWYiO2NvbG9yOmJsYWNrJz5Ub255PG86cD48 L286cD48L3NwYW4+PC9wPjxwIGNsYXNzPU1zb05vcm1hbD48c3BhbiBzdHls ZT0nZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseToiVGFob21hIiwic2Fu cy1zZXJpZiI7Y29sb3I6YmxhY2snPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFu PjwvcD48L2Rpdj48YnI+PGJyPgo8UCBhbGlnbj1jZW50ZXI+PEZPTlQgc3R5 bGU9IkJBQ0tHUk9VTkQtQ09MT1I6ICNmZmZmZmYiPgo8YnI+KioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKio8YnIgLz4KPGZvbnQgY29sb3I9IiMw MDk5MDAiIGZhY2U9IldlYmRpbmdzIiBzaXplPSI0Ij5QPC9mb250Pjxmb250 CiAgY29sb3I9IiMwMDk5MDAiIGZhY2U9InZlcmRhbmEsYXJpYWwsaGVsdmV0 aWNhIiBzaXplPSIyIj4gPHN0cm9uZz5QbGVhc2UKY29uc2lkZXIgdGhlIGVu dmlyb25tZW50IGJlZm9yZSBwcmludGluZyB0aGlzIGVtYWlsIG9yIGF0dGFj aG1lbnRzPC9zdHJvbmc+PC9mb250Pgo8UD4KPGJyPlRoaXMgZW1haWwgYW5k IGFueSBhdHRhY2htZW50cyBhcmUgY29uZmlkZW50aWFsLCBwcm90ZWN0ZWQg YnkgY29weXJpZ2h0IGFuZCBtYXkgYmUgbGVnYWxseSBwcml2aWxlZ2VkLiAg SWYgeW91IGFyZSBub3QgdGhlIGludGVuZGVkIHJlY2lwaWVudCwgdGhlbiB0 aGUgZGlzc2VtaW5hdGlvbiBvciBjb3B5aW5nIG9mIHRoaXMgZW1haWwgaXMg cHJvaGliaXRlZC4gSWYgeW91IGhhdmUgcmVjZWl2ZWQgdGhpcyBpbiBlcnJv ciwgcGxlYXNlIG5vdGlmeSB0aGUgc2VuZGVyIGJ5IHJlcGx5aW5nIGJ5IGVt YWlsIGFuZCB0aGVuIGRlbGV0ZSB0aGUgZW1haWwgY29tcGxldGVseSBmcm9t IHlvdXIgc3lzdGVtLiAgTmVpdGhlciBTcG9ydGluZyBJbmRleCBub3IgdGhl IHNlbmRlciBhY2NlcHRzIHJlc3BvbnNpYmlsaXR5IGZvciBhbnkgdmlydXMs IG9yIGFueSBvdGhlciBkZWZlY3Qgd2hpY2ggbWlnaHQgYWZmZWN0IGFueSBj b21wdXRlciBvciBJVCBzeXN0ZW0gaW50byB3aGljaCB0aGUgZW1haWwgaXMg cmVjZWl2ZWQgYW5kL29yIG9wZW5lZC4gIEl0IGlzIHRoZSByZXNwb25zaWJp bGl0eSBvZiB0aGUgcmVjaXBpZW50IHRvIHNjYW4gdGhlIGVtYWlsIGFuZCBu byByZXNwb25zaWJpbGl0eSBpcyBhY2NlcHRlZCBmb3IgYW55IGxvc3Mgb3Ig ZGFtYWdlIGFyaXNpbmcgaW4gYW55IHdheSBmcm9tIHJlY2VpcHQgb3IgdXNl IG9mIHRoaXMgZW1haWwuICBTcG9ydGluZyBJbmRleCBMdGQgaXMgYSBjb21w YW55IHJlZ2lzdGVyZWQgaW4gRW5nbGFuZCBhbmQgV2FsZXMgd2l0aCBjb21w YW55IG51bWJlciAyNjM2ODQyLCB3aG9zZSByZWdpc3RlcmVkIG9mZmljZSBp cyBhdCBHYXRld2F5IEhvdXNlLCBNaWx2ZXJ0b24gU3RyZWV0LCBMb25kb24s IFNFMTEgNEFQLiAgU3BvcnRpbmcgSW5kZXggTHRkIGlzIGF1dGhvcmlzZWQg YW5kIHJlZ3VsYXRlZCBieSB0aGUgVUsgRmluYW5jaWFsIFNlcnZpY2VzIEF1 dGhvcml0eSAocmVnLiBuby4gMTUwNDA0KSBhbmQgR2FtYmxpbmcgQ29tbWlz c2lvbiAocmVnLiBuby4gMDAwLTAyNzM0My1SLTMwODg5OC0wMDEpLiAgQW55 IGZpbmFuY2lhbCBwcm9tb3Rpb24gY29udGFpbmVkIGhlcmVpbiBoYXMgYmVl biBpc3N1ZWQKYW5kIGFwcHJvdmVkIGJ5IFNwb3J0aW5nIEluZGV4IEx0ZC48 YnIgLz4KPC9QPgo8YnI+T3V0Ym91bmQgZW1haWwgaGFzIGJlZW4gc2Nhbm5l ZCBmb3IgdmlydXNlcyBhbmQgU1BBTTxiciAvPgo8L0ZPTlQ+PC9QPgo8L2Jv ZHk+PC9odG1sPg== --_000_556325346CA26341B6F0530E07F90D96016C64CD967FGBGHEXCHCMS_--