Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9980BFB63 for ; Fri, 22 Mar 2013 04:51:15 +0000 (UTC) Received: (qmail 45811 invoked by uid 500); 22 Mar 2013 04:51:09 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 44575 invoked by uid 500); 22 Mar 2013 04:51:07 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 44343 invoked by uid 99); 22 Mar 2013 04:51:07 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Mar 2013 04:51:07 +0000 X-ASF-Spam-Status: No, hits=-10.8 required=5.0 tests=ENV_AND_HDR_SPF_MATCH,HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS,USER_IN_DEF_SPF_WL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of drusia@paypal.com designates 216.113.175.152 as permitted sender) Received: from [216.113.175.152] (HELO den-mipot-001.corp.ebay.com) (216.113.175.152) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Mar 2013 04:51:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paypal.com; i=@paypal.com; q=dns/txt; s=paypalcorp; t=1363927862; x=1395463862; h=from:to:subject:date:message-id:mime-version; bh=xHm6MdIRh6uwAp6YaWn+BLWr4aFy5mQSlg9kqtFoX0w=; b=DbOHKH/KLME4r/Sa3NAH4QkIUcN8Q/cmA4T7XL3pMoEVDtlgxgzc0E9B bX2N6pBt05B5Jr4HEG6G/Jo/DwlcqMNWtI9/dO+cGRt+U0II+Ucsj4Vlk Ni/Xs8vAh+hJEllHm+oyJoENkfHfZ7oanhgWibveKEHznl0zsBzvuo3rw Y=; X-EBay-Corp: Yes X-IronPort-AV: E=Sophos;i="4.84,890,1355126400"; d="scan'208,217";a="13927489" Received: from den-vtenf-002.corp.ebay.com (HELO DEN-EXMHT-005.corp.ebay.com) ([10.101.112.213]) by den-mipot-001.corp.ebay.com with ESMTP; 21 Mar 2013 21:50:41 -0700 Received: from PHX-EXMHT-004.corp.ebay.com (10.58.12.76) by DEN-EXMHT-005.corp.ebay.com (10.241.17.171) with Microsoft SMTP Server (TLS) id 14.2.318.4; Thu, 21 Mar 2013 22:50:41 -0600 Received: from PHX-EXRDA-S21.corp.ebay.com ([169.254.1.166]) by PHX-EXMHT-004.corp.ebay.com ([10.58.12.76]) with mapi id 14.02.0318.004; Thu, 21 Mar 2013 21:50:40 -0700 From: "Rusia, Devansh" To: "hdfs-dev@hadoop.apache.org" , "mapreduce-dev@hadoop.apache.org" , "user@hadoop.apache.org" Subject: FW: TupleWritable value in mapper Not getting cleaned up ( using CompositeInputFormat ) Thread-Topic: TupleWritable value in mapper Not getting cleaned up ( using CompositeInputFormat ) Thread-Index: Ac4lUt9sQK/KCILtSWib/zll3tt/LQAAHrUwAFlZKfA= Date: Fri, 22 Mar 2013 04:50:39 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.245.27.241] Content-Type: multipart/alternative; boundary="_000_E9A7DAFCFCA54D4E95387A55CF9B2DE90DB3E36FPHXEXRDAS21corp_" MIME-Version: 1.0 X-CFilter: Scanned X-Virus-Checked: Checked by ClamAV on apache.org --_000_E9A7DAFCFCA54D4E95387A55CF9B2DE90DB3E36FPHXEXRDAS21corp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, I am trying to do an outer join on to input files. But while joining the TupleWritable value in the mapper is not getting clea= ned up and so is using the previous values of a different key. The code I used is : ( 'plist' is containing the set of paths to be taken = as input ) jobConf.setInputFormat(CompositeInputFormat.class); jobConf.set("mapred.join.expr", CompositeInputFormat.compose(op, inputForma= tClass,plist.toArray(new Path[0]))); jobConf.setOutputFormat(outputFormatClass); inp1: anil1 10 anil2 20 anil3 30 dev1 40 dev2 50 inp2: anil1 100 dev1 400 dev2 500 dev3 600 outer join output: anil1 10,100 anil2 20,100 anil3 30,100 dev1 40,400 dev2 50,500 dev3 50,600 Actually It should be, right? anil1 10,100 anil2 20 anil3 30 dev1 40,400 dev2 50,500 dev3 600 Regards, Devansh Rusia --_000_E9A7DAFCFCA54D4E95387A55CF9B2DE90DB3E36FPHXEXRDAS21corp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

 

I am trying to do an outer join on to input files.

 

But while joining the TupleWritable value in the mapper is not getting cleaned up and so i= s using the previous values of a different key.

 

The code I used is : (  ‘plist’ is = containing the set of paths to be taken as input )

 

jobConf.setInput= Format(CompositeInputFormat.class);&nb= sp;            =  

jobConf.set("mapred.join.expr", CompositeInputFormat.compose(op, inputFormatClass,plist.toArray(new Path[0])));

jobConf.setOutputFormat(outputFormatClass);

 

inp1:

=  

= anil1     10

= anil2     20

= anil3     30

= dev1     40

= dev2     50

=  

= inp2:

=  

= anil1     100

= dev1     400

= dev2     500

= dev3     600

=  

=  

= outer join output:

=  

= anil1     10,100

= anil2     20,100

= anil3     30,100

= dev1     40,400

= dev2     50,500

= dev3     50,600

=  

= Actually It should be, right?

=  

anil1   &nbs= p; 10,100

anil2   &nbs= p; 20

anil3   &nbs= p; 30

dev1    = ; 40,400

dev2    = ; 50,500

dev3    = ; 600

 

Regards,

Devansh Rusia

--_000_E9A7DAFCFCA54D4E95387A55CF9B2DE90DB3E36FPHXEXRDAS21corp_--