Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8CE0D10179 for ; Fri, 18 Oct 2013 15:54:07 +0000 (UTC) Received: (qmail 94102 invoked by uid 500); 18 Oct 2013 15:54:00 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 93482 invoked by uid 500); 18 Oct 2013 15:53:55 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 93474 invoked by uid 99); 18 Oct 2013 15:53:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Oct 2013 15:53:54 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of sandy.ryza@cloudera.com designates 209.85.160.52 as permitted sender) Received: from [209.85.160.52] (HELO mail-pb0-f52.google.com) (209.85.160.52) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Oct 2013 15:53:48 +0000 Received: by mail-pb0-f52.google.com with SMTP id wy17so1063012pbc.39 for ; Fri, 18 Oct 2013 08:53:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=/H5+gCJK0VU6mf34jez1yfzvjn9hK2LBuKcvKaJrxio=; b=K7fYdDKPAaKrD7CyYk6JoNLV77XMVYfvq8Ff/cet3UGgWzbJ6oyQjv6IZdSyUcF1RS QibDHfYz3lkEO9PDFyajgi1TDjYBet3yrMN4v5RH+8SjUtXHZnMEUuRXd3T/L8pHgQer wEN1wqe88PIbe0E9I/2dFjDorRmdpkHVIWJk6sdIoo2F215xFKA/xS+b7efOOuMNDYRy Rodhwifao16+uSJYSbUhyRpivWNVD29gFI58RhkedT3q6qItm49/yYJKQ8gDRiluKOV7 pvW20213nJACnjKPKGL17yQ/+eC6o6VXlt6ddqAOu2Sru+0dhLwi7rcmqBE4fE+mzwil ihqg== X-Gm-Message-State: ALoCoQlnvR/EWx8VMmD/tNInkx0UGBQcXP2CJghEmiMkOQFQ7nd6Pw7R5klZ6e8qQGb+moh3X70a MIME-Version: 1.0 X-Received: by 10.66.7.68 with SMTP id h4mr4376601paa.0.1382111607378; Fri, 18 Oct 2013 08:53:27 -0700 (PDT) Received: by 10.70.52.2 with HTTP; Fri, 18 Oct 2013 08:53:27 -0700 (PDT) In-Reply-To: <1382108753.95655.YahooMailNeo@web141204.mail.bf1.yahoo.com> References: <1382108753.95655.YahooMailNeo@web141204.mail.bf1.yahoo.com> Date: Fri, 18 Oct 2013 08:53:27 -0700 Message-ID: Subject: Re: Yarn never use TeraSort#TotalOrderPartitioner when run TeraSort job? From: Sandy Ryza To: user@hadoop.apache.org, Ravi Prakash Content-Type: multipart/alternative; boundary=bcaec520e91965e7f804e905ef3f X-Virus-Checked: Checked by ClamAV on apache.org --bcaec520e91965e7f804e905ef3f Content-Type: text/plain; charset=ISO-8859-1 Hi Sam, Have you tried changing the map or reduce classes and seeing if that has any effect? -Sandy On Fri, Oct 18, 2013 at 8:05 AM, Ravi Prakash wrote: > Sam, I would guess that the jar file you think is running, is not actually > the one. I am guessing that in the task classpath, there is a normal jar > file (without your changes) which is being picked up before your modified > jar file. > > > > > On Thursday, October 17, 2013 10:13 PM, sam liu > wrote: > It's really weird and confusing me. Anyone can help this question? > > Thanks! > > > 2013/10/16 sam liu > > Hi Experts, > > In Hadoop-2.0.4, the TeraSort leverage TeraSort#TotalOrderPartitioner as > its Partitioner: 'job.setPartitionerClass(TotalOrderPartitioner.class);'. > However, seems Yarn did not execute the methods of > TeraSort#TotalOrderPartitioner at all. I did some tests to verify it as > below: > > Test 1: Add some code in the method readPartitions() and setConf() in > TeraSort#TotalOrderPartitioner to print some words and write some word to a > file. > Expected Result: Some words should be printed and wrote into a file > Actual Result: No word was printed and wrote into a file at all > > Test 2: Remove all existing methods in TeraSort#TotalOrderPartitioner, but > only remaining some necessary but empty methods in it > Expected Result: TeraSort job will ocurr some exception, as the specified > Partitioner is not implemented at all > Actual Result: TeraSort job completed successfully without any exception > > Above tests confused me a lot, because seems Yarn never use specified > partitioner TeraSort#TotalOrderPartitioner at all during job execution. > > Any one can help provide the reasons? > > Thanks very much! > > > > > --bcaec520e91965e7f804e905ef3f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Sam,

Have you tried changing t= he map or reduce classes and seeing if that has any effect?

-Sandy


=
On Fri, Oct 18, 2013 at 8:05 AM, Ravi Prakash <ravihoo@ymail.com> wrote:
Sam, I would guess that the jar f= ile you think is running, is not actually the one. I am guessing that in th= e task classpath, there is a normal jar file (without your changes) which i= s being picked up before your modified jar file.




On Thursday, October 17, 2013 10:1= 3 PM, sam liu <samliuhadoop@gmail.com> wrote:
It's really weird and confusing me. Anyone can help this question?

Thanks!


2013/10/16 sam liu <samli= uhadoop@gmail.com>
Hi Experts,

In Hadoop-2.0.= 4, the TeraSort leverage TeraSort#TotalOrderPartitioner as its Partitioner:= 'job.setPartitionerClass(TotalOrderPartitioner.class);'. However, = seems Yarn did not execute the methods of TeraSort#TotalOrderPartitioner at= all. I did some tests to verify it as below:

Test 1: Add some code in the method readPartitions= () and setConf() in TeraSort#TotalOrderPartitioner to print some words and = write some word to a file.
Expected Result: Some wo= rds should be printed and wrote into a file
Actual Result: No word was printed and wrote into a file at all

Test 2: Remove all existing methods i= n TeraSort#TotalOrderPartitioner, but only remaining some necessary but emp= ty methods in it
Expected Result: TeraSort job will ocurr some exception, as the specified P= artitioner is not implemented at all
Actual Result:= TeraSort job completed successfully without any exception

Above tests confused me a lot, because seems Yarn = never use specified partitioner TeraSort#TotalOrderPartitioner at all durin= g job execution.

Any one can help provide the reasons?

Thanks very much!


<= br> --bcaec520e91965e7f804e905ef3f--