kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张 明宝 <zhangmingbao2...@outlook.com>
Subject 答复: 答复: Coprocessor passed deadline! Maybe server is overloaded issue
Date Thu, 30 Aug 2018 04:03:08 GMT
Hi ShaoFeng:
         I will try all the methods you mentioned. thanks for your suggestions,

         Best wishes,
         MingBao Zhang,
________________________________
发件人: ShaoFeng Shi <shaofengshi@apache.org>
发送时间: 2018年8月29日 20:26
收件人: user
主题: Re: 答复: Coprocessor passed deadline! Maybe server is overloaded issue

Hi mingbao,

Your query is not a regular OLAP query. The "URL" cardinality is 20 million, and there is
no other filtering or limit condition exception the date, so its result set is still very
big. Besides, you use the bitmap for the distinct "deviceid", the bitmap is memory-consuming
which takes much memory than other measures. Totally it would take a lot of memory. That is
a big burden to IO and HBase region server.

Here are some suggestions:

1) if you only care about certain URLs which has many devices, you can consider to use the
Top N measure;
2) the "deviceid" is used in count distinct, you can remove it from dimension so the cube
can be smaller
3) try to normalize the URL to shorter length;
3) add more dimension and filter condition to the cube/query, so the resultset won't be that
big.


2018-08-30 10:10 GMT+08:00 张 明宝 <zhangmingbao2016@outlook.com<mailto:zhangmingbao2016@outlook.com>>:
Hi ShaoFeng:
       Thanks for your reply ! here are some fetures of data and cube。



My sql is:

select distinct url, count(distinct devid) as num_report from url_table where datetime ='2018-08-16'
group by urlorder by num_report desc


my data feature is like this:

columns count   count distinct
               data-lenth        type
datetime        100 million     300     10      string
devid   100 million
        1000    10      string

url     100 million
        20 million      0 - 500 string


And this is my Cube configuration:


[cid:5334574f-6671-4937-9b8a-c3485d47a522]

[cid:cab704a3-4131-4591-97f2-d9cdd7bcfabc]

[cid:c0aaac54-7005-44a4-bde0-74f52df5eaec]

________________________________
发件人: ShaoFeng Shi <shaofengshi@apache.org<mailto:shaofengshi@apache.org>>
发送时间: 2018年8月29日 18:47
收件人: user
主题: Re: Coprocessor passed deadline! Maybe server is overloaded issue

Hi Mingbao,

The timeout is 300 seconds, which is not configurable now. It can be externalized as a configuration
if needed.

The purpose of timeout is to prevent from too long waiting and resource occupation. As Kylin's
query should be returned in the second level, this threshold is long enough.

You need to investigate for the root cause. For example, whether the HBase cluster is in a
healthy state, what's the SQL query and whether it causes a bunch of scan and post-aggregation
in HBase, etc.

If you can share more information about the query and the Cube design, that would help in
the investigation.

2018-08-29 19:26 GMT+08:00 张 明宝 <zhangmingbao2016@outlook.com<mailto:zhangmingbao2016@outlook.com>>:
Hello,


       I built a cube with over 10 million source records successfully, and I could
use the cube to do some SQL query. However, when I want to do fuzzy query, I
have to face such errors:

##################################################################################################################
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException:
Coprocessor passed deadline! Maybe server is overloaded at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.checkDeadline(CubeVisitService.java:225)
at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:5555)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7873) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2008)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1990)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188) at org.apache.hadoop.hbase.ipc.Ca<http://org.apache.hadoop.hbase.ipc.Ca>llRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:748) while executing SQL: "select distinct dns, count(distinct
devid) as num_report from dns_table where datetime ='2018-08-16' group by dns order by num_report
desc LIMIT 50000"
  ##################################################################################################################
In kylin source code, this error is throwed by this method .

##############################################################
  private void checkDeadline(long deadline) throws DoNotRetryIOException {
        if (System.currentTimeMillis() > deadline) {
            logger.info<http://logger.info>("Deadline has passed, abort now!");
            throw new DoNotRetryIOException("Coprocessor passed deadline! Maybe server is
overloaded");
        }
    }
#############################################################

I want to set the deadline-value longer ,but i don't know how to change it and what negative
effects it will brings。


Looking forward to your reply and help.

Best wishes,
Mingbao Zhang



--
Best regards,

Shaofeng Shi 史少锋




--
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message