Return-Path: X-Original-To: apmail-spark-commits-archive@minotaur.apache.org Delivered-To: apmail-spark-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4DBE510871 for ; Thu, 23 Jan 2014 19:15:02 +0000 (UTC) Received: (qmail 18703 invoked by uid 500); 23 Jan 2014 19:14:59 -0000 Delivered-To: apmail-spark-commits-archive@spark.apache.org Received: (qmail 18661 invoked by uid 500); 23 Jan 2014 19:14:57 -0000 Mailing-List: contact commits-help@spark.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@spark.incubator.apache.org Delivered-To: mailing list commits@spark.incubator.apache.org Received: (qmail 18496 invoked by uid 99); 23 Jan 2014 19:14:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jan 2014 19:14:54 +0000 X-ASF-Spam-Status: No, hits=-2000.5 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 23 Jan 2014 19:14:53 +0000 Received: (qmail 16984 invoked by uid 99); 23 Jan 2014 19:14:30 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jan 2014 19:14:30 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 88E108BD66E; Thu, 23 Jan 2014 19:14:29 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: joshrosen@apache.org To: commits@spark.incubator.apache.org Date: Thu, 23 Jan 2014 19:14:36 -0000 Message-Id: <5e8b2a83485b4dfbbd0da0ae42407ed7@git.apache.org> In-Reply-To: <56aafa4783f747fe8887293068d82eb5@git.apache.org> References: <56aafa4783f747fe8887293068d82eb5@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [8/9] git commit: fixed ClassTag in mapPartitions X-Virus-Checked: Checked by ClamAV on apache.org fixed ClassTag in mapPartitions Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/60e74572 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/60e74572 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/60e74572 Branch: refs/heads/master Commit: 60e7457266eef18f562ef5cb93d62db1af821fdf Parents: 1442cd5 Author: eklavya Authored: Thu Jan 23 17:40:36 2014 +0530 Committer: eklavya Committed: Thu Jan 23 17:40:36 2014 +0530 ---------------------------------------------------------------------- .../org/apache/spark/api/java/JavaRDDLike.scala | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/60e74572/core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala b/core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala index 808c907..9680c6f 100644 --- a/core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala +++ b/core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala @@ -134,13 +134,21 @@ trait JavaRDDLike[T, This <: JavaRDDLike[T, This]] extends Serializable { /** * Return a new RDD by applying a function to each partition of this RDD. */ + def mapPartitions[U](f: FlatMapFunction[java.util.Iterator[T], U], preservesPartitioning: Boolean): JavaRDD[U] = { + def fn = (x: Iterator[T]) => asScalaIterator(f.apply(asJavaIterator(x)).iterator()) + JavaRDD.fromRDD(rdd.mapPartitions(fn, preservesPartitioning)(f.elementType()))(f.elementType()) + } + + /** + * Return a new RDD by applying a function to each partition of this RDD. + */ def mapPartitions(f: DoubleFlatMapFunction[java.util.Iterator[T]]): JavaDoubleRDD = { def fn = (x: Iterator[T]) => asScalaIterator(f.apply(asJavaIterator(x)).iterator()) new JavaDoubleRDD(rdd.mapPartitions(fn).map((x: java.lang.Double) => x.doubleValue())) } /** - * Return a new RDD by applying a function to each partition of this RDD. + * Return a new RDD by applying a function to each partition of this RDD. */ def mapPartitions[K2, V2](f: PairFlatMapFunction[java.util.Iterator[T], K2, V2]): JavaPairRDD[K2, V2] = { @@ -148,13 +156,6 @@ trait JavaRDDLike[T, This <: JavaRDDLike[T, This]] extends Serializable { JavaPairRDD.fromRDD(rdd.mapPartitions(fn))(f.keyType(), f.valueType()) } - /** - * Return a new RDD by applying a function to each partition of this RDD. - */ - def mapPartitions[U]( - f: FlatMapFunction[java.util.Iterator[T], U], preservesPartitioning: Boolean): JavaRDD[U] = { - rdd.mapPartitions[U]((x => f(asJavaIterator(x)).iterator), preservesPartitioning) - } /** * Return a new RDD by applying a function to each partition of this RDD.