Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 25192 invoked from network); 10 Jul 2008 22:16:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 10 Jul 2008 22:16:47 -0000 Received: (qmail 19032 invoked by uid 500); 10 Jul 2008 22:16:47 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 18990 invoked by uid 500); 10 Jul 2008 22:16:47 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 18977 invoked by uid 99); 10 Jul 2008 22:16:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jul 2008 15:16:47 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jul 2008 22:16:03 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 50A9323889FB; Thu, 10 Jul 2008 15:15:56 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r675784 - in /hadoop/core/trunk: CHANGES.txt src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/DataJoinReducerBase.java src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/TaggedMapOutput.java Date: Thu, 10 Jul 2008 22:15:56 -0000 To: core-commits@hadoop.apache.org From: cdouglas@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20080710221556.50A9323889FB@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: cdouglas Date: Thu Jul 10 15:15:55 2008 New Revision: 675784 URL: http://svn.apache.org/viewvc?rev=675784&view=rev Log: HADOOP-3684. Add a cloning function to the contrib/data_join framework permitting users to define a more efficient method for cloning values from the reduce than serialization/deserialization. Contributed by Runping Qi. Modified: hadoop/core/trunk/CHANGES.txt hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/DataJoinReducerBase.java hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/TaggedMapOutput.java Modified: hadoop/core/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=675784&r1=675783&r2=675784&view=diff ============================================================================== --- hadoop/core/trunk/CHANGES.txt (original) +++ hadoop/core/trunk/CHANGES.txt Thu Jul 10 15:15:55 2008 @@ -69,6 +69,10 @@ HADOOP-3660. Add replication factor for injecting blocks in simulated datanodes. (Sanjay Radia via cdouglas) + HADOOP-3684. Add a cloning function to the contrib/data_join framework + permitting users to define a more efficient method for cloning values from + the reduce than serialization/deserialization. (Runping Qi via cdouglas) + OPTIMIZATIONS HADOOP-3556. Removed lock contention in MD5Hash by changing the Modified: hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/DataJoinReducerBase.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/DataJoinReducerBase.java?rev=675784&r1=675783&r2=675784&view=diff ============================================================================== --- hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/DataJoinReducerBase.java (original) +++ hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/DataJoinReducerBase.java Thu Jul 10 15:15:55 2008 @@ -95,7 +95,6 @@ SortedMap retv = new TreeMap(); TaggedMapOutput aRecord = null; while (arg1.hasNext()) { - aRecord = (TaggedMapOutput) arg1.next(); this.numOfValues += 1; if (this.numOfValues % 100 == 0) { reporter.setStatus("key: " + key.toString() + " numOfValues: " @@ -104,13 +103,14 @@ if (this.numOfValues > this.maxNumOfValuesPerGroup) { continue; } - Text tag = new Text((Text)aRecord.getTag()); + aRecord = ((TaggedMapOutput) arg1.next()).clone(job); + Text tag = aRecord.getTag(); ResetableIterator data = retv.get(tag); if (data == null) { data = createResetableIterator(); retv.put(tag, data); } - data.add(WritableUtils.clone(aRecord, job)); + data.add(aRecord); } if (this.numOfValues > this.largestNumOfValues) { this.largestNumOfValues = numOfValues; Modified: hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/TaggedMapOutput.java URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/TaggedMapOutput.java?rev=675784&r1=675783&r2=675784&view=diff ============================================================================== --- hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/TaggedMapOutput.java (original) +++ hadoop/core/trunk/src/contrib/data_join/src/java/org/apache/hadoop/contrib/utils/join/TaggedMapOutput.java Thu Jul 10 15:15:55 2008 @@ -20,6 +20,8 @@ import org.apache.hadoop.io.Text; import org.apache.hadoop.io.Writable; +import org.apache.hadoop.io.WritableUtils; +import org.apache.hadoop.mapred.JobConf; /** * This abstract class serves as the base class for the values that @@ -46,5 +48,9 @@ } public abstract Writable getData(); + + public TaggedMapOutput clone(JobConf job) { + return (TaggedMapOutput) WritableUtils.clone(this, job); + } }