Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AD5BC1077C for ; Mon, 10 Jun 2013 13:16:39 +0000 (UTC) Received: (qmail 62247 invoked by uid 500); 10 Jun 2013 13:16:38 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 61874 invoked by uid 500); 10 Jun 2013 13:16:35 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 61865 invoked by uid 99); 10 Jun 2013 13:16:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jun 2013 13:16:34 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,UNRESOLVED_TEMPLATE X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ravimu@microsoft.com designates 207.46.163.158 as permitted sender) Received: from [207.46.163.158] (HELO na01-bn1-obe.outbound.protection.outlook.com) (207.46.163.158) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jun 2013 13:16:24 +0000 Received: from BY2FFO11FD009.protection.gbl (10.1.15.200) by BY2FFO11HUB029.protection.gbl (10.1.14.114) with Microsoft SMTP Server (TLS) id 15.0.707.0; Mon, 10 Jun 2013 13:16:01 +0000 Received: from TK5EX14HUBC107.redmond.corp.microsoft.com (131.107.125.37) by BY2FFO11FD009.mail.protection.outlook.com (10.1.14.73) with Microsoft SMTP Server (TLS) id 15.0.707.0 via Frontend Transport; Mon, 10 Jun 2013 13:16:01 +0000 Received: from CO9EHSOBE028.bigfish.com (157.54.51.112) by mail.microsoft.com (157.54.80.67) with Microsoft SMTP Server (TLS) id 14.3.136.1; Mon, 10 Jun 2013 13:15:18 +0000 Received: from mail208-co9-R.bigfish.com (10.236.132.245) by CO9EHSOBE028.bigfish.com (10.236.130.91) with Microsoft SMTP Server id 14.1.225.23; Mon, 10 Jun 2013 13:14:25 +0000 Received: from mail208-co9 (localhost [127.0.0.1]) by mail208-co9-R.bigfish.com (Postfix) with ESMTP id F00AE4200A7 for ; Mon, 10 Jun 2013 13:14:24 +0000 (UTC) X-Forefront-Antispam-Report-Untrusted: CIP:157.56.240.21;KIP:(null);UIP:(null);(null);H:BL2PRD0310HT005.namprd03.prod.outlook.com;R:internal;EFV:INT X-SpamScore: 3 X-BigFish: PS3(zz9371Ic85fhzz1f42h1ee6h1de0h1fdah1202h1e76h1d1ah1d2ah1fc6hzz17326ah18c673h8275bh8275dhz31h2a8h668h839hd24hf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh162dh1631h1758h18e1h1946h19b5h1ad9h1b0ah1bceh1d07h1d0ch1d2eh1d3fh1de9h1dfeh1dffh1e1dh17ej9a9j1155h) Received-SPF: softfail (mail208-co9: transitioning domain of microsoft.com does not designate 157.56.240.21 as permitted sender) client-ip=157.56.240.21; envelope-from=ravimu@microsoft.com; helo=BL2PRD0310HT005.namprd03.prod.outlook.com ;.outlook.com ; X-Forefront-Antispam-Report-Untrusted: SFV:SKI;SFS:;DIR:OUT;SFP:;SCL:-1;SRVR:BL2PR03MB594;H:BL2PR03MB593.namprd03.prod.outlook.com;LANG:en; Received: from mail208-co9 (localhost.localdomain [127.0.0.1]) by mail208-co9 (MessageSwitch) id 1370870063301531_25328; Mon, 10 Jun 2013 13:14:23 +0000 (UTC) Received: from CO9EHSMHS027.bigfish.com (unknown [10.236.132.244]) by mail208-co9.bigfish.com (Postfix) with ESMTP id 4718C38004F for ; Mon, 10 Jun 2013 13:14:23 +0000 (UTC) Received: from BL2PRD0310HT005.namprd03.prod.outlook.com (157.56.240.21) by CO9EHSMHS027.bigfish.com (10.236.130.37) with Microsoft SMTP Server (TLS) id 14.1.225.23; Mon, 10 Jun 2013 13:14:22 +0000 Received: from BL2PR03MB594.namprd03.prod.outlook.com (10.255.109.37) by BL2PRD0310HT005.namprd03.prod.outlook.com (10.255.97.40) with Microsoft SMTP Server (TLS) id 14.16.311.1; Mon, 10 Jun 2013 13:14:20 +0000 Received: from BL2PR03MB593.namprd03.prod.outlook.com (10.255.109.36) by BL2PR03MB594.namprd03.prod.outlook.com (10.255.109.37) with Microsoft SMTP Server (TLS) id 15.0.702.21; Mon, 10 Jun 2013 13:14:18 +0000 Received: from BL2PR03MB593.namprd03.prod.outlook.com ([169.254.3.144]) by BL2PR03MB593.namprd03.prod.outlook.com ([169.254.3.144]) with mapi id 15.00.0702.005; Mon, 10 Jun 2013 13:14:18 +0000 From: "Ravi Mummulla (BIG DATA)" To: "user@hive.apache.org" Subject: RE: Compression in Hive Thread-Topic: Compression in Hive Thread-Index: AQHOZaBzt82j5EuCwEWISM68spZKpZku7AVQ Date: Mon, 10 Jun 2013 13:14:17 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [2001:4898:23:5:7559:e1c8:47f5:7f46] Content-Type: multipart/alternative; boundary="_000_e39bc10227f04942996faa1cb2138d48BL2PR03MB593namprd03pro_" MIME-Version: 1.0 X-OrganizationHeadersPreserved: BL2PR03MB594.namprd03.prod.outlook.com X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% X-FOPE-CONNECTOR: Id%59$Dn%HIVE.APACHE.ORG$RO%2$TLS%6$FQDN%corpf5vips-237160.customer.frontbridge.com$TlsDn% X-CrossPremisesHeadersPromoted: TK5EX14HUBC107.redmond.corp.microsoft.com X-CrossPremisesHeadersFiltered: TK5EX14HUBC107.redmond.corp.microsoft.com X-Forefront-Antispam-Report: CIP:131.107.125.37;CTRY:US;IPV:CAL;IPV:NLI;EFV:NLI;SFV:NSPM;SFS:(199002)(189002)(377454002)(47446002)(16236675002)(44976003)(76482001)(50986001)(33646001)(74876001)(6806003)(31966008)(74502001)(69226001)(56816003)(76786001)(71186001)(81542001)(77096001)(74706001)(56776001)(74366001)(47736001)(74316001)(79102001)(20776003)(53806001)(49866001)(63696002)(15202345002)(80022001)(4396001)(54356001)(76576001)(77982001)(51856001)(47976001)(74662001)(76796001)(16676001)(59766001)(54316002)(46102001)(512954002)(81342001)(65816001)(24736002)(3826001);DIR:OUT;SFP:;SCL:1;SRVR:BY2FFO11HUB029;H:TK5EX14HUBC107.redmond.corp.microsoft.com;CLIP:131.107.125.37;RD:InfoDomainNonexistent;A:1;MX:1;LANG:en; X-OriginatorOrg: microsoft.onmicrosoft.com X-Forefront-PRVS: 087396016C X-Virus-Checked: Checked by ClamAV on apache.org --_000_e39bc10227f04942996faa1cb2138d48BL2PR03MB593namprd03pro_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Documentation is here https://cwiki.apache.org/confluence/display/Hive/Comp= ressedStorage. Performance overhead is trivial for larger amounts of data b= ut may be magnified as data size gets smaller. Typically where you gain is = data transfers between nodes and disk reads/writes. Again, the larger the d= ata size the more the gain. Thanks. From: Sachin Sudarshana [mailto:sachin.hadoop@gmail.com] Sent: Sunday, June 9, 2013 11:04 PM To: user@hive.apache.org Subject: Compression in Hive Hi, I have been testing the usefulness of compression in Hive. I have a general= question, I would like to know if there are any particular cases where compression in= hive can actually prove useful while running any MR jobs. Any pointers/examples would really be useful! Thank you, Sachin --_000_e39bc10227f04942996faa1cb2138d48BL2PR03MB593namprd03pro_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Documentation is here https://cwiki.apache.org/confluence/display/Hive/CompressedStora= ge. Performance overhead is trivial for larger amounts of data but may be magnified as data size ge= ts smaller. Typically where you gain is data transfers between nodes and di= sk reads/writes. Again, the larger the data size the more the gain.

 <= /p>

Thanks.=

 <= /p>

From: Sachin= Sudarshana [mailto:sachin.hadoop@gmail.com]
Sent: Sunday, June 9, 2013 11:04 PM
To: user@hive.apache.org
Subject: Compression in Hive

 

Hi,

 

I have been testing the usefulness of compression in= Hive. I have a general question,

 

I would like to know if there are any particular cas= es where compression in hive can actually prove useful while running any MR= jobs.

 

Any pointers/examples would really be useful!

 

Thank you,

Sachin

 

--_000_e39bc10227f04942996faa1cb2138d48BL2PR03MB593namprd03pro_--