spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From juliuszsompolski <...@git.apache.org>
Subject [GitHub] spark pull request #20624: [SPARK-23445] ColumnStat refactoring
Date Sun, 25 Feb 2018 20:13:14 GMT
Github user juliuszsompolski commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20624#discussion_r170465448
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
    @@ -387,6 +390,101 @@ case class CatalogStatistics(
       }
     }
     
    +/**
    + * This class of statistics for a column is used in [[CatalogTable]] to interact with
metastore.
    + */
    +case class CatalogColumnStat(
    +  distinctCount: Option[BigInt] = None,
    +  min: Option[String] = None,
    +  max: Option[String] = None,
    +  nullCount: Option[BigInt] = None,
    +  avgLen: Option[Long] = None,
    +  maxLen: Option[Long] = None,
    +  histogram: Option[Histogram] = None) {
    +
    +  /**
    +   * Returns a map from string to string that can be used to serialize the column stats.
    +   * The key is the name of the column and name of the field (e.g. "colName.distinctCount"),
    +   * and the value is the string representation for the value.
    +   * min/max values are stored as Strings. They can be deserialized using
    +   * [[ColumnStat.fromExternalString]].
    --- End diff --
    
    I think that actually everything from ColumnStat object should move.
    `fromExternalString` / `toExternalString` -> `CatalogColumnStat`
    
    And also:
    `supportsDatatype` / `supportsHistogram` -> `AnalyzeColumnCommand`
    `statExprs` / `rowToColumnStat` -> `AnalyzeColumnCommand`
    because they are tied to that specific method of stats collection.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message