orc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From omalley <...@git.apache.org>
Subject [GitHub] orc pull request #292: ORC-203 - Update StringStatistics to trim long string...
Date Wed, 25 Jul 2018 16:42:39 GMT
Github user omalley commented on a diff in the pull request:

    https://github.com/apache/orc/pull/292#discussion_r205181230
  
    --- Diff: java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java ---
    @@ -543,35 +549,87 @@ public void reset() {
           super.reset();
           minimum = null;
           maximum = null;
    +      isLowerBoundSet = false;
    +      isUpperBoundSet = false;
           sum = 0;
         }
     
         @Override
         public void updateString(Text value) {
           if (minimum == null) {
    -        maximum = minimum = new Text(value);
    +        if(value.toString().length() > MAX_STRING_LENGTH_RECORDED) {
    +         minimum = new Text(
    --- End diff --
    
    We can even use the structure of UTF-8 to do truncateLowerBound without forming a string.
In particular you want to search from 1024 to find the first byte of a character (0xxx xxxx
or 11xx xxxx) and truncate just before it.


---

Mime
View raw message