kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lemont (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-3416) Kylin bitmap null pointer exception
Date Sun, 01 Jul 2018 16:11:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529142#comment-16529142
] 

Lemont commented on KYLIN-3416:
-------------------------------

[~Shaofengshi]

I'm sorry i am the novice of kylin,I don't know how calcite explain the sql.I will provide
the message for you and try to fix it at a later time.

To make it easy ,I will use the KYLIN_SALES table.

The simple sql:

select count(distinct KYLIN_SALES.PRICE) from KYLIN_SALES  group by KYLIN_SALES.LEAF_CATEG_ID/100

I think the case of this error is exactAggregation is true, cuboid id is 1

the exactAggregation case the bitmap only have the size of self but don't have the bitmap
value

The model definition is:
{
  "uuid": "d4a843df-f135-4038-9d54-f3241ddc491c",
  "last_modified": 1530459283091,
  "version": "2.4.0.20500",
  "name": "kylin_sales_model_testnull",
  "owner": "ADMIN",
  "is_draft": false,
  "description": "",
  "fact_table": "DEFAULT.KYLIN_SALES",
  "lookups": [],
  "dimensions": [
    \{
      "table": "KYLIN_SALES",
      "columns": [
        "TRANS_ID",
        "SELLER_ID",
        "BUYER_ID",
        "PART_DT",
        "LEAF_CATEG_ID",
        "LSTG_FORMAT_NAME",
        "LSTG_SITE_ID",
        "OPS_USER_ID",
        "OPS_REGION"
      ]
    }
  ],
  "metrics": [
    "KYLIN_SALES.PRICE",
    "KYLIN_SALES.ITEM_COUNT"
  ],
  "filter_condition": "",
  "partition_desc": \{
    "partition_date_column": null,
    "partition_time_column": null,
    "partition_date_start": 1325376000000,
    "partition_date_format": "yyyy-MM-dd",
    "partition_time_format": "HH:mm:ss",
    "partition_type": "APPEND",
    "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
  },
  "capacity": "MEDIUM"
}
The cube definition :
{
  "uuid": "b88422cf-2cbf-4cd3-921b-73a927fd09aa",
  "last_modified": 1530459405127,
  "version": "2.4.0.20500",
  "name": "test_bitmap_null_cube",
  "is_draft": false,
  "model_name": "kylin_sales_model_testnull",
  "description": "",
  "null_string": null,
  "dimensions": [
    \{
      "name": "LEAF_CATEG_ID",
      "table": "KYLIN_SALES",
      "column": "LEAF_CATEG_ID",
      "derived": null
    }
  ],
  "measures": [
    \{
      "name": "_COUNT_",
      "function": {
        "expression": "COUNT",
        "parameter": {
          "type": "constant",
          "value": "1"
        },
        "returntype": "bigint"
      }
    },
    \{
      "name": "PRICE",
      "function": {
        "expression": "COUNT_DISTINCT",
        "parameter": {
          "type": "column",
          "value": "KYLIN_SALES.PRICE"
        },
        "returntype": "bitmap"
      }
    }
  ],
  "dictionaries": [
    \{
      "column": "KYLIN_SALES.PRICE",
      "builder": "org.apache.kylin.dict.GlobalDictionaryBuilder"
    }
  ],
  "rowkey": \{
    "rowkey_columns": [
      {
        "column": "KYLIN_SALES.LEAF_CATEG_ID",
        "encoding": "dict",
        "encoding_version": 1,
        "isShardBy": false
      }
    ]
  },
  "hbase_mapping": \{
    "column_family": [
      {
        "name": "F1",
        "columns": [
          {
            "qualifier": "M",
            "measure_refs": [
              "_COUNT_"
            ]
          }
        ]
      },
      \{
        "name": "F2",
        "columns": [
          {
            "qualifier": "M",
            "measure_refs": [
              "PRICE"
            ]
          }
        ]
      }
    ]
  },
  "aggregation_groups": [
    \{
      "includes": [
        "KYLIN_SALES.LEAF_CATEG_ID"
      ],
      "select_rule": {
        "hierarchy_dims": [],
        "mandatory_dims": [],
        "joint_dims": []
      }
    }
  ],
  "signature": "TVE29H3H4ElAm/4pEljkOQ==",
  "notify_list": [],
  "status_need_notify": [
    "ERROR",
    "DISCARDED",
    "SUCCEED"
  ],
  "partition_date_start": 0,
  "partition_date_end": 3153600000000,
  "auto_merge_time_ranges": [
    604800000,
    2419200000
  ],
  "volatile_range": 0,
  "retention_range": 0,
  "engine_type": 2,
  "storage_type": 2,
  "override_kylin_properties": {},
  "cuboid_black_list": [],
  "parent_forward": 3,
  "mandatory_dimension_set_list": [],
  "snapshot_table_desc_list": []
}

> Kylin bitmap null pointer exception
> -----------------------------------
>
>                 Key: KYLIN-3416
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3416
>             Project: Kylin
>          Issue Type: Bug
>          Components: Query Engine
>    Affects Versions: v2.3.1
>            Reporter: Lemont
>            Priority: Blocker
>
> Hi,team:
>   Ithink there is a conflict between dimensional aggregation and count distinct.
> For example:
> select
>  (1524931200 - biz_time)/(30*86400),
>  count(DISTINCT id) id
>  from test
> where pt ='20180621'
>  group by (1524931200 - biz_time)/(30*86400)
> Caused by: java.lang.NullPointerException
>  at org.apache.kylin.measure.bitmap.RoaringBitmapCounter.getMutableBitmap(RoaringBitmapCounter.java:58)
>  at org.apache.kylin.measure.bitmap.RoaringBitmapCounter.orWith(RoaringBitmapCounter.java:72)
>  at org.apache.kylin.measure.bitmap.BitmapAggregator.aggregate(BitmapAggregator.java:43)
>  at org.apache.kylin.measure.bitmap.BitmapDistinctCountAggFunc.add(BitmapDistinctCountAggFunc.java:31)
>  
> The problem cased by  GTCubeStorageQueryBase.isNeedStorageAggregation
> There are three dimension int the cube,the sql use cuboid 5 and the cuboid.getColumns
is
> pt,bize_time.
> The groupD is bize_time and singleValueD is pt.
> So isNeedStorageAggregation return false.
> But in fact this sql need storage agregation because the group is (1524931200 - biz_time)/(30*86400)
not only biz_time 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message