Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B154C113AD for ; Thu, 17 Jul 2014 07:55:05 +0000 (UTC) Received: (qmail 1772 invoked by uid 500); 17 Jul 2014 07:54:55 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 1660 invoked by uid 500); 17 Jul 2014 07:54:55 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 1650 invoked by uid 99); 17 Jul 2014 07:54:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jul 2014 07:54:54 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of prabakaran.1.natarajan@nsn.com designates 93.183.12.31 as permitted sender) Received: from [93.183.12.31] (HELO demumfd002.nsn-inter.net) (93.183.12.31) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jul 2014 07:54:51 +0000 Received: from demuprx017.emea.nsn-intra.net ([10.150.129.56]) by demumfd002.nsn-inter.net (8.14.3/8.14.3) with ESMTP id s6H7sEVv010035 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Thu, 17 Jul 2014 07:54:15 GMT Received: from SGSIHTC003.nsn-intra.net ([10.159.225.20]) by demuprx017.emea.nsn-intra.net (8.12.11.20060308/8.12.11) with ESMTP id s6H7rPdW024622 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Thu, 17 Jul 2014 09:54:13 +0200 Received: from SGSIHTC006.nsn-intra.net (10.159.225.23) by SGSIHTC003.nsn-intra.net (10.159.225.20) with Microsoft SMTP Server (TLS) id 14.3.181.6; Thu, 17 Jul 2014 15:52:32 +0800 Received: from SGSIMBX001.nsn-intra.net ([169.254.1.40]) by SGSIHTC006.nsn-intra.net ([10.159.225.23]) with mapi id 14.03.0181.006; Thu, 17 Jul 2014 15:52:32 +0800 From: "Natarajan, Prabakaran 1. (NSN - IN/Bangalore)" To: "user@hadoop.apache.org" Subject: Multiple Part files Thread-Topic: Multiple Part files Thread-Index: Ac+hlA+3OIBb0nxPRDSgnQozRc7vVA== Date: Thu, 17 Jul 2014 07:52:31 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.159.225.122] Content-Type: multipart/alternative; boundary="_000_DD08C21E8C680641B67C6C279273A5350F0AC928SGSIMBX001nsnin_" MIME-Version: 1.0 X-purgate-type: clean X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de X-purgate: clean X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate-size: 2450 X-purgate-ID: 151667::1405583655-00007A71-2D58CFCB/0/0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_DD08C21E8C680641B67C6C279273A5350F0AC928SGSIMBX001nsnin_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi After Map Reduce job, we are seeing multiple small part files in the output= directory. We are using RC file format (snappy codec) 1) Do each part file will take 64MB block size? 2) How to merge these multiple RC format part files into one RC file? 3) What is the pros-cons of having multiple part files? 4) Do merging part files will improve performance? Thanks and Regards Prabakaran.N aka NP nsn, Bangalore When "I" is replaced by "We" - even Illness becomes "Wellness" --_000_DD08C21E8C680641B67C6C279273A5350F0AC928SGSIMBX001nsnin_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hi
 
After Map Reduce job, we are seeing multiple small part files in the o= utput directory. We are using RC file format (snappy codec)
 
  1. Do each part file will take 64MB block size?
  2. How to merge these= multiple RC format part files into one RC file?
  3. What is the pros-c= ons of having multiple part files?
  4. Do merging part files will impro= ve performance?
 
Thanks and Regards
Prabakaran.N  aka NP
nsn= , Bangalore <= /span>
<= i>When "I" is replaced by "We" - even Illness becomes &= quot;Wellness"
 
 
 
 
 --_000_DD08C21E8C680641B67C6C279273A5350F0AC928SGSIMBX001nsnin_--