spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-26297) improve the doc of Distribution/Partitioning
Date Tue, 11 Dec 2018 00:17:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-26297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715835#comment-16715835
] 

ASF GitHub Bot commented on SPARK-26297:
----------------------------------------

cloud-fan commented on a change in pull request #23249: [SPARK-26297][SQL] improve the doc
of Distribution/Partitioning
URL: https://github.com/apache/spark/pull/23249#discussion_r240431621
 
 

 ##########
 File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala
 ##########
 @@ -118,10 +115,12 @@ case class HashClusteredDistribution(
 
 /**
  * Represents data where tuples have been ordered according to the `ordering`
- * [[Expression Expressions]].  This is a strictly stronger guarantee than
- * [[ClusteredDistribution]] as an ordering will ensure that tuples that share the
- * same value for the ordering expressions are contiguous and will never be split across
- * partitions.
+ * [[Expression Expressions]]. Its requirement is defined as the following:
+ *   - Given any 2 adjacent partitions, all the rows of the second partition must be larger
than or
+ *     equal to any row in the first partition, according to the `ordering` expressions.
 
 Review comment:
   @hvanhovell We need this relaxed requirement, otherwise we have to remove the optimization
[here](https://github.com/apache/spark/pull/23249/files#diff-14e5db1a500000efdaf9c760aeb5ced4R273)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> improve the doc of Distribution/Partitioning
> --------------------------------------------
>
>                 Key: SPARK-26297
>                 URL: https://issues.apache.org/jira/browse/SPARK-26297
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Wenchen Fan
>            Assignee: Wenchen Fan
>            Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message