Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EADDD9207 for ; Thu, 14 Jun 2012 15:56:21 +0000 (UTC) Received: (qmail 10003 invoked by uid 500); 14 Jun 2012 15:56:20 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 9919 invoked by uid 500); 14 Jun 2012 15:56:20 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 9910 invoked by uid 99); 14 Jun 2012 15:56:20 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jun 2012 15:56:20 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [204.154.183.83] (HELO smtp1.exar.com) (204.154.183.83) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jun 2012 15:56:13 +0000 Received: from FRE1-CAS2.hq.exar.com (10.127.1.158) by FRE1-CAS.hq.exar.com (10.127.1.162) with Microsoft SMTP Server (TLS) id 8.1.436.0; Thu, 14 Jun 2012 08:55:52 -0700 Received: from FRE1EXCH02.hq.exar.com ([fe80::e9a7:6160:4e13:1677]) by FRE1-CAS2.hq.exar.com ([fe80::60c0:dc64:7883:c86%10]) with mapi; Thu, 14 Jun 2012 08:55:51 -0700 From: Tim Broberg To: "mapreduce-user@hadoop.apache.org" Date: Thu, 14 Jun 2012 08:55:34 -0700 Subject: RE: codec compression ratio Thread-Topic: codec compression ratio Thread-Index: Ac1KCR5c0oPXF3O4QHyTaa4Mx8ZjOgAPQNx3 Message-ID: References: <03AF7A47D43C954E92AC1F7CD5C124020C3103FE@TLN-MBX1.corp.seven.com> In-Reply-To: <03AF7A47D43C954E92AC1F7CD5C124020C3103FE@TLN-MBX1.corp.seven.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Have you considered deflate or bzip? - Tim. ________________________________________ From: Marek Miglinski [mmiglinski@seven.com] Sent: Thursday, June 14, 2012 1:39 AM To: mapreduce-user@hadoop.apache.org Subject: codec compression ratio When procession 65billion records and using LZO or Snappy codecs, disk IO i= s at 100% because mappers are spilling all the time, but CPU is at 40%. Is = there a setting where I can raise compression ratio for map/reduce internal= temp data (for LZO or Snappy)? So that I can raise effort on CPU and lower= IO? Google didn't gave any ideas... Thanks. Marek M. The information contained in this email is intended only for the personal a= nd confidential use of the recipient(s) named above. The information and a= ny attached documents contained in this message may be Exar confidential an= d/or legally privileged. If you are not the intended recipient, you are he= reby notified that any review, use, dissemination or reproduction of this m= essage is strictly prohibited and may be unlawful. If you have received th= is communication in error, please notify us immediately by return email and= delete the original message.