kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Long (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-3686) Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the Web UI defaults to ID_HBASE and provides no safeguards against storage type mismatch
Date Tue, 04 Dec 2018 12:20:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708623#comment-16708623
] 

Chao Long commented on KYLIN-3686:
----------------------------------

Hi Vsevolod, The front-end will get the value of parameter "kylin.storage.default"(default
ID_SHARDED_HBASE=2) from the kylin.properties and will use it as the cube's storage type.
Can you check this parameter in your kylin.properties?

Another situation is the front-end does not get this value for some unknown reason, it will
pass a null value to the back-end, and the back-end will use ID_HBASE as the default value
of storage type.

If the latter, [KYLIN-3636|https://issues.apache.org/jira/browse/KYLIN-3636] has fixed it
and will release soon.

> Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the Web UI defaults
to ID_HBASE and provides no safeguards against storage type mismatch
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KYLIN-3686
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3686
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Measure - TopN, Metadata, Web 
>    Affects Versions: v2.5.0
>         Environment: HDP 2.5.6, Kylin 2.5
>            Reporter: Vsevolod Ostapenko
>            Assignee: Shaofeng SHI
>            Priority: Major
>             Fix For: v2.6.0
>
>
> When new cube is defined via Kylin 2.5 UI, the default cube storage type is set to 0
(ID_HBASE).
>  Top_N metric support is currently hard coded to expect cube storage type 2 (ID_SHARDED_HBASE),
and it *_does not_* check if the cube storage type is the "sharded HBASE".
>  UI provides no safeguards either to prevent a user from defining a cube with Top_N metric
that would blow up on the cube building stage with a perplexing stack trace like the following:
> {quote}2018-10-22 16:15:50,388 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
>  java.lang.ArrayIndexOutOfBoundsException
>  at java.lang.System.arraycopy(Native Method)
>  at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
>  at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
>  at org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
>  at org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
>  at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> {quote}
> Please, either:
> – modify Top_N code to support all cube storage types (not only ID_SHARDED_HBASE),
>  or 
>  – modify Top_N code to perform explicit check for cube storage type and raise descriptive
exception, when cube storage is not the one that is expected. Plus update the UI to prevent
the user from creating cube definitions that are incompatible with the storage type compatible
with Top_N measure
> PS: NDCCuboidBuilder,java contains the following line:
> {quote}int offset = RowConstants.ROWKEY_SHARDID_LEN + RowConstants.ROWKEY_CUBOIDID_LEN;
// skip shard and cuboidId{quote}
> If cube storage type is not ID_SHARDED_HBASE, offset is calculated incorrectly, which
leads to ArrayIndexOutOfBounds exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message