Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 93189 invoked from network); 2 Jul 2008 03:20:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 2 Jul 2008 03:20:56 -0000 Received: (qmail 96746 invoked by uid 500); 2 Jul 2008 03:20:53 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 96708 invoked by uid 500); 2 Jul 2008 03:20:53 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 96697 invoked by uid 99); 2 Jul 2008 03:20:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jul 2008 20:20:53 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates 216.139.236.158 as permitted sender) Received: from [216.139.236.158] (HELO kuber.nabble.com) (216.139.236.158) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jul 2008 03:20:02 +0000 Received: from isper.nabble.com ([192.168.236.156]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1KDstG-0005R3-Lz for core-user@hadoop.apache.org; Tue, 01 Jul 2008 20:20:18 -0700 Message-ID: <18229533.post@talk.nabble.com> Date: Tue, 1 Jul 2008 20:20:18 -0700 (PDT) From: novice user To: core-user@hadoop.apache.org Subject: Re: Combiner is optional though it is specified? In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: pallavip.05@gmail.com References: <18213887.post@talk.nabble.com> X-Virus-Checked: Checked by ClamAV on apache.org Regarding the conclusion, I am parsing the inputs in combiner and reducer differently. For example the output value of mapper is "s:d" where as the output value of combiner is "s,d". So, in reducer, I am assuming the input as "s,d" and trying to parse it. There I got the exception because it got input as "s:d". I am using hadoop-17. Icouldn't get exactly what you meant by no guarantee on the number of times a combiner is run. Can you please elaborate a bit on this? Thanks Arun C Murthy-2 wrote: > > > On Jul 1, 2008, at 4:04 AM, novice user wrote: > >> >> Hi all, >> I have a query regarding the functionality of combiner. >> Is it possible to ignore combiner code for some of the outputs of >> mapper and >> directly being sent to reducer though combiner is specified in job >> configuration? >> Because, I figured out that, when I am running on large amounts of >> data, >> some of the mapper output is directly reached reducer. I am >> wondering how >> can this be possible when I have specified combiner in the job >> configuration. Can any one please let me know if this thing happens? >> > > Can you elaborate on how you reached the conclusion that the output > of some maps isn't going through the combiner? > > Also, what version of hadoop are you using? hadoop-0.18 onwards there > aren't guarantees on the number of times a combiner is run... > > Arun > >> >> >> -- >> View this message in context: http://www.nabble.com/Combiner-is- >> optional-though-it-is-specified--tp18213887p18213887.html >> Sent from the Hadoop core-user mailing list archive at Nabble.com. >> > > > -- View this message in context: http://www.nabble.com/Combiner-is-optional-though-it-is-specified--tp18213887p18229533.html Sent from the Hadoop core-user mailing list archive at Nabble.com.