spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cloud-fan <>
Subject [GitHub] spark pull request: [SPARK-12480][SQL] add Hash expression that ca...
Date Wed, 23 Dec 2015 12:59:31 GMT
Github user cloud-fan commented on a diff in the pull request:
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala
    @@ -114,6 +114,7 @@ private[sql] object StatFunctions extends Logging {
           if (element == null) "null" else element.toString
         // get the distinct values of column 2, so that we can make them the column names
    +    // ???
         val distinctCol2: Map[Any, Int] =
  => cleanElement(e.get(1))).distinct.zipWithIndex.toMap
    --- End diff --
    The `counts` is the result of aggregation without sort, so the order of distinct values
of column 2 is non-deterministic. This means, when we use `distinctCol2` to generate the field
names of output schema, the order of field names is also non-deterministic.
    When I changed the hash function in `Exchange`, one crosstable test failed because the
field names don't match. I simply updated that test, need someone to confirm this is the right
thing to do.
    cc @rxin 

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message