From reviews-return-1124945-archive-asf-public=cust-asf.ponee.io@spark.apache.org Sun Jul 5 11:57:05 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 22B88180652 for ; Sun, 5 Jul 2020 13:57:05 +0200 (CEST) Received: (qmail 65630 invoked by uid 500); 5 Jul 2020 11:57:04 -0000 Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@spark.apache.org Received: (qmail 65618 invoked by uid 99); 5 Jul 2020 11:57:04 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2020 11:57:04 +0000 From: =?utf-8?q?GitBox?= To: reviews@spark.apache.org Subject: =?utf-8?q?=5BGitHub=5D_=5Bspark=5D_maropu_commented_on_a_change_in_pull_requ?= =?utf-8?q?est_=2328988=3A_=5BSPARK-32163=5D=5BSQL=5D_Nested_pruning_should_?= =?utf-8?q?work_even_with_cosmetic_variations?= Message-ID: <159395022442.29655.17904726171673754052.asfpy@gitbox.apache.org> Date: Sun, 05 Jul 2020 11:57:04 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit In-Reply-To: References: maropu commented on a change in pull request #28988: URL: https://github.com/apache/spark/pull/28988#discussion_r449868437 ########## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala ########## @@ -174,9 +176,12 @@ object NestedColumnAliasing { // If all nested fields of `attr` are used, we don't need to introduce new aliases. // By default, ColumnPruning rule uses `attr` already. + // Note that we need to remove cosmetic variations first, so we only count a + // nested field once. if (nestedFieldToAlias.nonEmpty && - nestedFieldToAlias - .map { case (nestedField, _) => totalFieldNum(nestedField.dataType) } + dedupNestedFields.map(_.canonicalized.asInstanceOf[ExtractValue]) Review comment: nit: we don't need the cast here? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org For additional commands, e-mail: reviews-help@spark.apache.org