spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jose-torres <...@git.apache.org>
Subject [GitHub] spark pull request #21560: [SPARK-24386][SS] coalesce(1) aggregates in conti...
Date Mon, 25 Jun 2018 20:22:12 GMT
Github user jose-torres commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21560#discussion_r197929262
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
    @@ -349,6 +349,17 @@ object UnsupportedOperationChecker {
                   _: DeserializeToObject | _: SerializeFromObject | _: SubqueryAlias |
                   _: TypedFilter) =>
             case node if node.nodeName == "StreamingRelationV2" =>
    +        case Repartition(1, false, _) =>
    +        case node: Aggregate =>
    +          val aboveSinglePartitionCoalesce = node.find {
    +            case Repartition(1, false, _) => true
    +            case _ => false
    +          }.isDefined
    +
    +          if (!aboveSinglePartitionCoalesce) {
    --- End diff --
    
    I agree that it wouldn't be needed, but partitioning information is not always available
during analysis. So I don't think we can write the more granular check suggested here.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message