spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-18522) Create explicit contract for column stats serialization
Date Mon, 21 Nov 2016 07:50:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15682815#comment-15682815
] 

Apache Spark commented on SPARK-18522:
--------------------------------------

User 'rxin' has created a pull request for this issue:
https://github.com/apache/spark/pull/15959

> Create explicit contract for column stats serialization
> -------------------------------------------------------
>
>                 Key: SPARK-18522
>                 URL: https://issues.apache.org/jira/browse/SPARK-18522
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>            Reporter: Reynold Xin
>            Assignee: Reynold Xin
>
> The current implementation of column stats uses the base64 encoding of the internal UnsafeRow
format to persist statistics (in table properties in Hive metastore). This is an internal
format that is not stable across different versions of Spark and should NOT be used for persistence.
> In addition, it would be better if statistics stored in the catalog is human readable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message