orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From omalley <...@git.apache.org>
Subject [GitHub] orc pull request #292: ORC-203 - Update StringStatistics to trim long string...
Date Wed, 25 Jul 2018 16:42:39 GMT
Github user omalley commented on a diff in the pull request:

    https://github.com/apache/orc/pull/292#discussion_r205180384
  
    --- Diff: java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java ---
    @@ -517,10 +519,14 @@ public int hashCode() {
     
       protected static final class StringStatisticsImpl extends ColumnStatisticsImpl
           implements StringColumnStatistics {
    +    public static final int MAX_STRING_LENGTH_RECORDED = 1024;
    --- End diff --
    
    We should probably use the number of bytes instead of the number of characters. I'd propose
that we always truncate a whole number of characters, with a maximum of 1024 bytes.
    
    For ASCII strings, it will have the same result, but larger characters may end up at 1021
to 1024 bytes long.


---

Mime
View raw message