Return-Path: X-Original-To: apmail-spark-commits-archive@minotaur.apache.org Delivered-To: apmail-spark-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0EFEC17329 for ; Thu, 9 Oct 2014 22:03:04 +0000 (UTC) Received: (qmail 3936 invoked by uid 500); 9 Oct 2014 22:03:03 -0000 Delivered-To: apmail-spark-commits-archive@spark.apache.org Received: (qmail 3908 invoked by uid 500); 9 Oct 2014 22:03:03 -0000 Mailing-List: contact commits-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list commits@spark.apache.org Received: (qmail 3899 invoked by uid 99); 9 Oct 2014 22:03:03 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Oct 2014 22:03:03 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id 99F12DAF1; Thu, 9 Oct 2014 22:03:03 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: marmbrus@apache.org To: commits@spark.apache.org Message-Id: <86029966595646e485fc6df53e22e42a@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: git commit: [SPARK-3858][SQL] Pass the generator alias into logical plan node Date: Thu, 9 Oct 2014 22:03:03 +0000 (UTC) Repository: spark Updated Branches: refs/heads/master 0c0e09f56 -> bc3b6cb06 [SPARK-3858][SQL] Pass the generator alias into logical plan node The alias parameter is being ignored, which makes it more difficult to specify a qualifier for Generator expressions. Author: Nathan Howell Closes #2721 from NathanHowell/SPARK-3858 and squashes the following commits: 8aa0f43 [Nathan Howell] [SPARK-3858][SQL] Pass the generator alias into logical plan node Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bc3b6cb0 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bc3b6cb0 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bc3b6cb0 Branch: refs/heads/master Commit: bc3b6cb06153d6b05f311dd78459768b6cf6a404 Parents: 0c0e09f Author: Nathan Howell Authored: Thu Oct 9 15:03:01 2014 -0700 Committer: Michael Armbrust Committed: Thu Oct 9 15:03:01 2014 -0700 ---------------------------------------------------------------------- sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala | 2 +- .../src/test/scala/org/apache/spark/sql/DslQuerySuite.scala | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/bc3b6cb0/sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala b/sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala index 594bf8f..948122d 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala @@ -360,7 +360,7 @@ class SchemaRDD( join: Boolean = false, outer: Boolean = false, alias: Option[String] = None) = - new SchemaRDD(sqlContext, Generate(generator, join, outer, None, logicalPlan)) + new SchemaRDD(sqlContext, Generate(generator, join, outer, alias, logicalPlan)) /** * Returns this RDD as a SchemaRDD. Intended primarily to force the invocation of the implicit http://git-wip-us.apache.org/repos/asf/spark/blob/bc3b6cb0/sql/core/src/test/scala/org/apache/spark/sql/DslQuerySuite.scala ---------------------------------------------------------------------- diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DslQuerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DslQuerySuite.scala index d001abb..45e58af 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/DslQuerySuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/DslQuerySuite.scala @@ -147,6 +147,14 @@ class DslQuerySuite extends QueryTest { (1, 1, 1, 2) :: Nil) } + test("SPARK-3858 generator qualifiers are discarded") { + checkAnswer( + arrayData.as('ad) + .generate(Explode("data" :: Nil, 'data), alias = Some("ex")) + .select("ex.data".attr), + Seq(1, 2, 3, 2, 3, 4).map(Seq(_))) + } + test("average") { checkAnswer( testData2.groupBy()(avg('a)), --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org For additional commands, e-mail: commits-help@spark.apache.org