carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sounakr <...@git.apache.org>
Subject [GitHub] carbondata pull request #1079: [WIP]Measure Filter implementation
Date Sun, 02 Jul 2017 20:11:09 GMT
Github user sounakr commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/1079#discussion_r125191345
  
    --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
    @@ -395,6 +440,58 @@ public static DimColumnFilterInfo getNoDictionaryValKeyMemberForFilter(
       }
     
       /**
    +   * This method will get the no dictionary data based on filters and same
    +   * will be in ColumnFilterInfo
    +   *
    +   * @param evaluateResultListFinal
    +   * @param isIncludeFilter
    +   * @return ColumnFilterInfo
    +   */
    +  public static ColumnFilterInfo getMeasureValKeyMemberForFilter(
    +      List<String> evaluateResultListFinal, boolean isIncludeFilter, DataType dataType,
    +      CarbonMeasure carbonMeasure) throws FilterUnsupportedException {
    +    List<byte[]> filterValuesList = new ArrayList<byte[]>(20);
    +    String result = null;
    +    try {
    +      int length = evaluateResultListFinal.size();
    +      for (int i = 0; i < length; i++) {
    +        result = evaluateResultListFinal.get(i);
    +        if (CarbonCommonConstants.MEMBER_DEFAULT_VAL.equals(result)) {
    +          filterValuesList.add(new byte[0]);
    +          continue;
    +        }
    +        // TODO have to understand what method to be used for measures.
    +        // filterValuesList
    +        //  .add(DataTypeUtil.getBytesBasedOnDataTypeForNoDictionaryColumn(result, dataType));
    +
    +        filterValuesList
    +            .add(DataTypeUtil.getMeasureByteArrayBasedOnDataTypes(result, dataType, carbonMeasure));
    +
    +      }
    +    } catch (Throwable ex) {
    +      throw new FilterUnsupportedException("Unsupported Filter condition: " + result,
ex);
    +    }
    +
    +    Comparator<byte[]> filterMeasureComaparator = new Comparator<byte[]>()
{
    +
    +      @Override public int compare(byte[] filterMember1, byte[] filterMember2) {
    +        // TODO Auto-generated method stub
    +        return ByteUtil.UnsafeComparer.INSTANCE.compareTo(filterMember1, filterMember2);
    --- End diff --
    
    We are converting String dataTypes into Bytes array and then saving into filterValueList.
From populateFilterResolvedInfo itself we convert all dataType to strings and then pass it
along. We may have to rectify and pass on actual datatype from populateFilterResolvedInfo.

    
    But all filterValue Comparision in measures are currently sequential, so there no chance
of getting a wrong result. i.e. in Include, RowLevelLessThan, RowLevelLessThanEqual, RowLevelGrtThanEqual,
RowLevelGrtThan. Also Range is not implemented yet for measures where 2 filter values should
be in ascending order.  In the next stage optimization we can hold the filter values in actual
datatype and have comparator for each. This was comparision will be proper and we dont have
to convert to datatype to byte and again back to object while doing actual comparision, we
can carry object all along.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message