kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaofeng SHI (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-3636) in kylin 2.4.1 and 2.5.1 topn aggregation query caused exception
Date Thu, 15 Nov 2018 13:42:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688073#comment-16688073
] 

Shaofeng SHI commented on KYLIN-3636:
-------------------------------------

Hi Kang-sen,

 

MR or Spark engine is configured in the "Advanced setting" step when create a Cube, with the
label "Engine type". In the Cube json, it is "engine_type", 2 means MR, 4 means Spark.

 

Regarding the documentation, which page did you follow? is it [https://kylin.apache.org/docs/tutorial/create_cube.html?|https://kylin.apache.org/docs/tutorial/create_cube.html] 

> in kylin 2.4.1 and 2.5.1 topn aggregation query caused exception
> ----------------------------------------------------------------
>
>                 Key: KYLIN-3636
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3636
>             Project: Kylin
>          Issue Type: Bug
>          Components: Query Engine
>    Affects Versions: v2.4.1
>            Reporter: KANG-SEN LU
>            Assignee: Shaofeng SHI
>            Priority: Major
>             Fix For: v2.6.0
>
>
> Hi, ShaoFeng:
>  
> Thanks for the reply. I missed this email and not responded earlier, I am sorry.
>  
> I tried to reproduce this problem with the sample database, and it did not happen.
>  
> So I am hoping by collecting enough "clue", someone can figure out why this problem occurred.
>  
> --------------------------
> I issued the following query at the sample project to exercise the topn aggregation:
>  
> select seller_id, SUM(price) as total from kylin_sales group by seller_id order by total
limit 5;
>  
> With my own added debugging, I saw the following log in the kylin.log: (the query worked
OK).
>  
> 2018-10-16 16:18:19,963 INFO  [Query a747f16f-4b12-cc97-08d2-9b45c27a529f-90] model.FunctionDesc:59
: KSL22222, getRewriteFieldName=_KY_SUM_KYLIN_SALES_PRICE_
> 2018-10-16 16:18:19,963 INFO  [kylin-coproc--pool12-t1] v2.CubeHBaseEndpointRPC:217
: Query-a747f16f-4b12-cc97-08d2-9b45c27a529f: send request to the init region server anovadata4.anovadata.local
on table ANOVA_KYLIN_25X_K758MEAWJG
> 2018-10-16 16:18:19,963 INFO  [Query a747f16f-4b12-cc97-08d2-9b45c27a529f-90] topn.TopNMeasureType:399
: KSL888: in TopNMeasureType.java, sumFieldName= _KY_SUM_KYLIN_SALES_PRICE_
>  
>  
> When I was executing my project query, I issued the following select statement:
>  
> SELECT  ZETTICSDW.A_VL_HOURLY_V.IMSIID \"ZETTICSDW_A_VL_HOURLY_V_IMSIID\", SUM(ZETTICSDW.A_VL_HOURLY_V.SIG_EVENT_COUNT)
\"vl_aggs_model___USERS_BY_ERROR_3XX\"  FROM  ZETTICSDW.A_VL_HOURLY_V inner JOIN ZETTICSDW.T_VL_TRANSACTION_RULE_V
ON ( ZETTICSDW.A_VL_HOURLY_V.CAUSE_CODE_KEY = ZETTICSDW.T_VL_TRANSACTION_RULE_V.CAUSE_CODE_KEY
AND ZETTICSDW.A_VL_HOURLY_V.REASON_CODE_KEY = ZETTICSDW.T_VL_TRANSACTION_RULE_V.REASON_CODE_KEY
AND ZETTICSDW.A_VL_HOURLY_V.TRANSACTION_TYPE_KEY = ZETTICSDW.T_VL_TRANSACTION_RULE_V.TRANSACTION_TYPE_KEY) 
> WHERE  ((ZETTICSDW.A_VL_HOURLY_V.THEDATE = '20180209') AND ((ZETTICSDW.A_VL_HOURLY_V.THEHOUR
>= '02') AND (ZETTICSDW.A_VL_HOURLY_V.THEHOUR <= '03'))) AND ZETTICSDW.T_VL_TRANSACTION_RULE_V.DISPLAY_STRING
LIKE '%+3%'  
> GROUP BY  ZETTICSDW.A_VL_HOURLY_V.IMSIID  
> ORDER BY  \"vl_aggs_model___USERS_BY_ERROR_3XX\"
> LIMIT 25
>  
> An exception occurred within the method "private ColumnRowType buildColumnRowType()"
of "query/src/main/java/org/apache/kylin/query/relnode/OLAPTableScan.java".
>  
>         if (columns.size() != rowType.getFieldCount()) {
>             throw new IllegalStateException("RowType=" + rowType.getFieldCount()
+ ", ColumnRowType=" + columns.size());
>         }
>  
> It printed "RowType=133, ColumnRowType=132".
> The RowType list contains one extra column name: "ANY _KY_SUM_1_3a1aedef_SIG_EVENT_COUNT_".
>  
>  
> I think this has something to do with the bug fix "KYLIN-3359 Support sum(expression)
if possible".
>  
> After this bug fix was submitted, I noticed that a lot of column name was added into
rowType like "_KY_SUM_XXX".
>  
> This strange column name "_KY_SUM_1_3a1aedef_SIG_EVENT_COUNT_" is very similar.
>  
> I also found that this extra column name, only existed in RowType but not in ColumnRowType,
was added in the method "public void implementRewrite(RewriteImplementor implementor)" within
"OLAPJoinRel.java".
>  
> With my own debug statement, I saw this debug text in kylin.log:
>  
> relnode.OLAPJoinRel:362 : KSL54: newField= #132: _KY_SUM_1_6735969a_SIG_EVENT_COUNT_
ANY
>  
>  
> I hope someone with a deeper technical knowledge in kylin query engine can figure out
what was causing the problem I have seen.
>  
> Thanks again.
>  
> Kang-sen
>  
> *From:* ShaoFeng Shi [[mailto:shaofengshi@apache.org]] 
>  *Sent:* Friday, October 05, 2018 9:59 PM
>  *To:* user <[user@kylin.apache.org|mailto:user@kylin.apache.org]>
>  *Subject:* Re: any body see topn in kylin 2.5.1 working?
>  
> Hi Kang-Sen,
>  
> Didn't see this; Can you reproduce the problem with the sample cube? 
>  
> Kang-Sen Lu <[klu@anovadata.com|mailto:klu@anovadata.com]> 于2018年10月6日周六
上午12:21写道:
> I have been running kylin 2.2.1 with topn aggregation OK.
>  
> Recently, I upgraded to kylin 2.4.1 and 2.5.1, in both releases I am having query problem
related to topn support.
>  
> The problem is an exception occurred with the following error message in kylin.log:
>  
> Caused by: java.lang.IllegalStateException: RowType=133, ColumnRowType=132
>         at org.apache.kylin.query.relnode.OLAPJoinRel.buildColumnRowType(OLAPJoinRel.java:223)
>         at org.apache.kylin.query.relnode.OLAPJoinRel.implementRewrite(OLAPJoinRel.java:359)
>         at org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
>         at org.apache.kylin.query.relnode.OLAPFilterRel.implementRewrite(OLAPFilterRel.java:163)
>         at org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
>         at org.apache.kylin.query.relnode.OLAPProjectRel.implementRewrite(OLAPProjectRel.java:228)
>         at org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
>         at org.apache.kylin.query.relnode.OLAPAggregateRel.implementRewrite(OLAPAggregateRel.java:402)
>         at org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
>         at org.apache.kylin.query.relnode.OLAPSortRel.implementRewrite(OLAPSortRel.java:86)
>         at org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
>         at org.apache.kylin.query.relnode.OLAPLimitRel.implementRewrite(OLAPLimitRel.java:109)
>         at org.apache.kylin.query.relnode.OLAPRel$RewriteImplementor.visitChild(OLAPRel.java:174)
>         at org.apache.kylin.query.relnode.OLAPToEnumerableConverter.implement(OLAPToEnumerableConverter.java:100)
>         at org.apache.calcite.adapter.enumerable.EnumerableRelImplementor.implementRoot(EnumerableRelImplementor.java:108)
>         at org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:92)
>         at org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1281)
>         at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:331)
>         at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:228)
>         at org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:801)
>         at org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:656)
>         at org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:618)
>         at org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:214)
>         at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:603)
>         at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
>         at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
>         ... 84 more
>  
>  
> --
> Best regards,
>  
> Shaofeng Shi 史少锋
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message