kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: kylin help
Date Fri, 27 Apr 2018 09:09:29 GMT
Thank you Ma Gang for sharing the troubleshooting experiences!


2018-04-27 15:29 GMT+08:00 旭鹤 <xuhe@dianjia.io>:

>
>          help me  look at  下面的这个错误?
>
>
> 2018-04-26 12:57:34,266 INFO  [Query 05ce476e-bced-4f4a-
> 953e-9083df2e2909-85] service.QueryService:893 : Setting
> current statement's max rows to 0
> 2018-04-26 12:57:38,622 ERROR [Query 9786f329-b6ce-42c2-
> a365-8da8e9e62f10-51] service.QueryService:536 : Exception
> while executing query
> java.sql.SQLException: Error while executing SQL "select
>        COALESCE(SUM(total_sale_money_kpi),0) as total_sale_money_kpi
>       , COALESCE(SUM(total_ord_count_kpi),0) as total_ord_count_kpi
>       , COALESCE(SUM(total_sale_count_kpi),0) as total_sale_count_kpi
>   from view_dm_brand_dc_storage_sale_dm_v2 a  inner join dim_date d on a.
> start_day = d.date_id
>    WHERE 1=1
>     and brand_id = 10091
>       and a.start_day >= '2018-04-01'
>       and a.start_day <= '2018-04-25'
>      and a.storage_id = 254120473799950338
>      and a.storage_id in
>      (
>       254120473799950338
>      )": Timeout visiting cube! Check why coprocessor
> exception is not sent back? In coprocessor Self-
> termination is checked every 100 scanned rows, the
> configured timeout(15000) cannot support this many scans?
>  at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
>  at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
>  at org.apache.calcite.avatica.AvaticaStatement.executeInternal(
> AvaticaStatement.java:156)
>  at org.apache.calcite.avatica.AvaticaStatement.
> executeQuery(AvaticaStatement.java:218)
>  at org.apache.kylin.rest.service.QueryService.execute(
> QueryService.java:931)
>  at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(
> QueryService.java:661)
>  at org.apache.kylin.rest.service.QueryService.query(
> QueryService.java:188)
>  at org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(
> QueryService.java:496)
>  at org.apache.kylin.rest.service.QueryService.
> doQueryWithCache(QueryService.java:455)
>  at org.apache.kylin.rest.service.QueryService.
> doQueryWithCache(QueryService.java:390)
>  at org.apache.kylin.rest.controller.QueryController.
> query(QueryController.java:86)
>  at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498)
>  at org.springframework.web.method.support.InvocableHandlerMethod.
> doInvoke(InvocableHandlerMethod.java:205)
>  at org.springframework.web.method.support.InvocableHandlerMethod.
> invokeForRequest(InvocableHandlerMethod.java:133)
>  at org.springframework.web.servlet.mvc.method.annotation.
> ServletInvocableHandlerMethod.invokeAndHandle(
> ServletInvocableHandlerMethod.java:97)
>  at org.springframework.web.servlet.mvc.method.annotation.
> RequestMappingHandlerAdapter.invokeHandlerMethod(
> RequestMappingHandlerAdapter.java:827)
>  at org.springframework.web.servlet.mvc.method.annotation.
> RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.
> java:738)
>  at org.springframework.web.servlet.mvc.method.
> AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>  at org.springframework.web.servlet.DispatcherServlet.
> doDispatch(DispatcherServlet.java:967)
>  at org.springframework.web.servlet.DispatcherServlet.
> doService(DispatcherServlet.java:901)
>  at org.springframework.web.servlet.FrameworkServlet.processRequest(
> FrameworkServlet.java:970)
>  at org.springframework.web.servlet.FrameworkServlet.
> doPost(FrameworkServlet.java:872)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
>  at org.springframework.web.servlet.FrameworkServlet.
> service(FrameworkServlet.java:846)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
>  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:303)
>  at org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:208)
>  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
>  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:241)
>  at org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:208)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:317)
>  at org.springframework.security.web.access.intercept.
> FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
>  at org.springframework.security.web.access.intercept.
> FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.access.ExceptionTranslationFilter.
> doFilter(ExceptionTranslationFilter.java:114)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.session.SessionManagementFilter.
> doFilter(SessionManagementFilter.java:137)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.authentication.
> AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.
> java:111)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.servletapi.
> SecurityContextHolderAwareRequestFilter.doFilter(
> SecurityContextHolderAwareRequestFilter.java:170)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.
> doFilter(RequestCacheAwareFilter.java:63)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.authentication.
> www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.
> java:215)
>  at org.springframework.web.filter.OncePerRequestFilter.
> doFilter(OncePerRequestFilter.java:107)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.authentication.
> AbstractAuthenticationProcessingFilter.doFilter(
> AbstractAuthenticationProcessingFilter.java:200)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.authentication.
> logout.LogoutFilter.doFilter(LogoutFilter.java:116)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.header.HeaderWriterFilter.
> doFilterInternal(HeaderWriterFilter.java:64)
>  at org.springframework.web.filter.OncePerRequestFilter.
> doFilter(OncePerRequestFilter.java:107)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.context.request.async.
> WebAsyncManagerIntegrationFilter.doFilterInternal(
> WebAsyncManagerIntegrationFilter.java:56)
>  at org.springframework.web.filter.OncePerRequestFilter.
> doFilter(OncePerRequestFilter.java:107)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.context.
> SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilt
> er.java:105)
>  at org.springframework.security.web.FilterChainProxy$
> VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at org.springframework.security.web.FilterChainProxy.doFilterInternal(
> FilterChainProxy.java:214)
>  at org.springframework.security.web.FilterChainProxy.
> doFilter(FilterChainProxy.java:177)
>  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(
> DelegatingFilterProxy.java:346)
>  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(
> DelegatingFilterProxy.java:262)
>  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:241)
>  at org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:208)
>  at com.thetransactioncompany.cors.CORSFilter.doFilter(
> CORSFilter.java:209)
>  at com.thetransactioncompany.cors.CORSFilter.doFilter(
> CORSFilter.java:244)
>  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:241)
>  at org.apache.catalina.core.ApplicationFilterChain.doFilter(
> ApplicationFilterChain.java:208)
>  at org.apache.catalina.core.StandardWrapperValve.invoke(
> StandardWrapperValve.java:219)
>  at org.apache.catalina.core.StandardContextValve.invoke(
> StandardContextValve.java:110)
>  at org.apache.catalina.core.StandardHostValve.invoke(
> StandardHostValve.java:169)
>  at org.apache.catalina.valves.ErrorReportValve.
> invoke(ErrorReportValve.java:103)
>  at org.apache.catalina.valves.AccessLogValve.invoke(
> AccessLogValve.java:962)
>  at org.apache.catalina.core.StandardEngineValve.invoke(
> StandardEngineValve.java:116)
>  at org.apache.catalina.connector.CoyoteAdapter.
> service(CoyoteAdapter.java:445)
>  at org.apache.coyote.http11.AbstractHttp11Processor.process(
> AbstractHttp11Processor.java:1115)
>  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.
> process(AbstractProtocol.java:637)
>  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(
> NioEndpoint.java:1775)
>  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(
> NioEndpoint.java:1734)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
>  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(
> TaskThread.java:61)
>  at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Timeout visiting cube! Check why
> coprocessor exception is not sent back? In coprocessor
> Self-termination is checked every 100 scanned rows, the
> configured timeout(15000) cannot support this many scans?
>  at org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(
> ExpectedSizeIterator.java:70)
>  at org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(
> ExpectedSizeIterator.java:32)
>  at com.google.common.collect.TransformedIterator.next(
> TransformedIterator.java:48)
>  at org.apache.kylin.storage.gtrecord.SortMergedPartitionResultItera
> tor.hasNext(SortMergedPartitionResultIterator.java:70)
>  at com.google.common.collect.Iterators$PeekingImpl.hasNext(
> Iterators.java:1216)
>  at org.apache.kylin.gridtable.GTStreamAggregateScanner$
> AbstractStreamMergeIterator.hasNext(GTStreamAggregateScanner.java:80)
>  at org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(
> SegmentCubeTupleIterator.java:149)
>  at com.google.common.collect.Iterators$5.hasNext(Iterators.java:596)
>  at org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(
> SequentialCubeTupleIterator.java:139)
>  at org.apache.kylin.query.enumerator.OLAPEnumerator.
> moveNext(OLAPEnumerator.java:65)
>  at Baz$1$1.moveNext(Unknown Source)
>  at org.apache.calcite.linq4j.EnumerableDefaults.aggregate(
> EnumerableDefaults.java:116)
>  at org.apache.calcite.linq4j.DefaultEnumerable.aggregate(
> DefaultEnumerable.java:107)
>  at Baz.bind(Unknown Source)
>  at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(
> CalcitePrepare.java:335)
>  at org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(
> CalciteConnectionImpl.java:294)
>  at org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(
> CalciteMetaImpl.java:559)
>  at org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(
> CalciteMetaImpl.java:550)
>  at org.apache.calcite.avatica.AvaticaResultSet.
> execute(AvaticaResultSet.java:204)
>  at org.apache.calcite.jdbc.CalciteResultSet.execute(
> CalciteResultSet.java:67)
>  at org.apache.calcite.jdbc.CalciteResultSet.execute(
> CalciteResultSet.java:44)
>  at org.apache.calcite.avatica.AvaticaConnection$1.
> execute(AvaticaConnection.java:630)
>  at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(
> CalciteMetaImpl.java:607)
>  at org.apache.calcite.avatica.AvaticaConnection.
> prepareAndExecuteInternal(AvaticaConnection.java:638)
>  at org.apache.calcite.avatica.AvaticaStatement.executeInternal(
> AvaticaStatement.java:149)
>  ... 83 more
>
> ------------------------------------------------------------------
> 发件人:Ma Gang <mg4work@163.com>
> 发送时间:2018年4月25日(星期三) 10:15
> 收件人:user <user@kylin.apache.org>; 旭鹤 <xuhe@dianjia.io>
> 主 题:Re:回复 kylin help
>
> According to log, the coprocessor executor service is in 'Shutting down'
> state, so all coprocessor requests are rejected. Is this log happens when
> you shutdown the Kylin server? From code I cannot see anywhere that
> coprocessor executor will be shut down, except in the system's shutdown
> hook.
> You should monitor the hbase region server metrics, this looks like region
> server issue.
>
> 在 2018-04-24 20:00:21,"旭鹤" <xuhe@dianjia.io> 写道:
>     设置的队列大小是500
>     发生问题的时候,都在等待。
>
> 看下这段能不能看出来什么
>
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@4ca122b1 rejected from
> java.util.concurrent.ThreadPoolExecutor@466b92cd[Shutting down, pool size
> = 31, active threads = 31, queued tasks = 0, completed tasks = 52306]
>     at java.util.concurrent.ThreadPoolExecutor$
> AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
>     at java.util.concurrent.ThreadPoolExecutor.reject(
> ThreadPoolExecutor.java:830)
>     at java.util.concurrent.ThreadPoolExecutor.execute(
> ThreadPoolExecutor.java:1379)
>     at java.util.concurrent.AbstractExecutorService.submit(
> AbstractExecutorService.java:134)
>     at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.
> java:1795)
>     at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.
> runEPRange(CubeHBaseEndpointRPC.java:205)
>     at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.access$
> 000(CubeHBaseEndpointRPC.java:69)
>     at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(
> CubeHBaseEndpointRPC.java:186)
>     at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748)
> 2018-04-24 19:11:29,755 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> common.KylinConfig:278 : KYLIN_CONF property was not set, will seek
> KYLIN_HOME env variable
> 2018-04-24 19:11:29,756 ERROR [pool-10-thread-283]
> v2.CubeHBaseEndpointRPC:340 : <sub-thread for Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f
> GTScanRequest 2257581>Error when visiting cubes by endpoint
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@4ca122b1 rejected from
> java.util.concurrent.ThreadPoolExecutor@466b92cd[Shutting down, pool size
> = 31, active threads = 31, queued tasks = 0, completed tasks = 52306]
>     at java.util.concurrent.ThreadPoolExecutor$
> AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
>     at java.util.concurrent.ThreadPoolExecutor.reject(
> ThreadPoolExecutor.java:830)
>     at java.util.concurrent.ThreadPoolExecutor.execute(
> ThreadPoolExecutor.java:1379)
>     at java.util.concurrent.AbstractExecutorService.submit(
> AbstractExecutorService.java:134)
>     at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.
> java:1795)
>     at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.
> runEPRange(CubeHBaseEndpointRPC.java:205)
>     at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC.access$
> 000(CubeHBaseEndpointRPC.java:69)
>     at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(
> CubeHBaseEndpointRPC.java:186)
>     at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748)
> 2018-04-24 19:11:29,756 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> gtrecord.SequentialCubeTupleIterator:73 : Using Iterators.concat to merge
> segment results
> 2018-04-24 19:11:29,756 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> enumerator.OLAPEnumerator:120 : return TupleIterator...
> 2018-04-24 19:11:29,756 ERROR [Query efb08523-db3b-434b-b4f7-35ad
>
>
> env variable
> 2018-04-24 19:11:29,724 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> gtrecord.CubeSegmentScanner:56 : Init CubeSegmentScanner for segment
> 20180420000000_20180421000000
> 2018-04-24 19:11:29,725 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 60000 ms, use 54000 ms as timeout
> for coprocessor
> 2018-04-24 19:11:29,725 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> v2.CubeHBaseEndpointRPC:152 : Serialized scanRequestBytes 2778 bytes,
> rawScanBytesString 88 bytes
> 2018-04-24 19:11:29,726 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> v2.CubeHBaseEndpointRPC:154 : The scan 589bd9fa for segment
> view_dm_brand_dc_storage_sale_dm_model_v2_cube[20180420000000_20180421000000]
> is as below with 1 separate raw scans, shard part of start/end key is set
> to 0
> 2018-04-24 19:11:29,726 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_X0VZKW9TP4: cuboid require
> post aggregation, from 56 to 63 Start: \x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x3F\x1E\x04\x28\x00\x00\x00\x00 (\x00\x00\x00\x00\x00\x00\x00\
> x00\x00?\x1E\x04(\x00\x00\x00\x00) Stop: \x00\x00\x00\x00\x00\x00\x00\
> x00\x00\x3F\x1E\x04\x28\x00\xFF\xFF\xFF\x00 (\x00\x00\x00\x00\x00\x00\x00\
> x00\x00?\x1E\x04(\x00\xFF\xFF\xFF\x00) Fuzzy key counts: 1. Fuzzy keys :
> \x00\x00\x00\x00\x00\x00\x00\x00\x00\x3F\x1E\x04\x28\x00\x00\x00\x00
> \x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x01\x01\x01;
> 2018-04-24 19:11:29,727 DEBUG [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> v2.CubeHBaseEndpointRPC:159 : Submitting rpc to 3 shards starting from
> shard 2, scan range count 1
> 2018-04-24 19:11:29,727 INFO [Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f-49]
> common.KylinConfig:319 : Loading kylin-defaults.properties from
> file:/opt/apache-kylin-2.3.1-bin/tomcat/webapps/kylin/WEB-
> INF/lib/kylin-core-common-2.3.1.jar!/kylin-defaults.properties
> 2018-04-24 19:11:29,727 ERROR [pool-10-thread-283]
> v2.CubeHBaseEndpointRPC:340 : <sub-thread for Query efb08523-db3b-434b-b4f7-35ad7d3f0d6f
> GTScanRequest 7b499b30>Error when visiting cubes by endpoint
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@4ca122b1 rejected from
> java.util.concurrent.ThreadPoolExecutor@466b92cd[Shutting down, pool size
> = 31, active threads = 31, queued tasks = 0, completed tasks = 52306]
>     at java.util.concurrent.ThreadPoolExecutor$
> AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
>     at java.util.concurrent.ThreadPoolExecutor.reject(
> ThreadPoolExecutor.java:830)
>
>
> v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 60000 ms, use 54000 ms as
> timeout for coprocessor
>
> 协处理器处理超时
>
>
>
> ------------------------------------------------------------------
> 发件人:旭鹤 <xuhe@dianjia.io>
> 发送时间:2018年4月24日(星期二) 18:54
> 收件人:Ma Gang <mg4work@163.com>
> 主 题:回复:回复:kylin help
>
> 这个里有很多在等待
>
>
> ------------------------------------------------------------------
> 发件人:Ma Gang <mg4work@163.com>
> 发送时间:2018年4月24日(星期二) 18:25
> 收件人:旭鹤 <xuhe@dianjia.io>
> 主 题:Re:回复:kylin help
>
> That may be the cause, you should monitor the hbase metrics properly(rpc
> queue size etc.)
>
> At 2018-04-24 17:56:28, "旭鹤" <xuhe@dianjia.io> wrote:
> Hi,
> 我们刚刚调了hbase 的 hbase.ipc.server.read.threadpool.size
> 由原来的10 调至1000 目前还在测试当中
> 你看看,有没有相关的联系。
>
> ------------------------------------------------------------------
> 发件人:Ma Gang <mg4work@163.com>
> 发送时间:2018年4月24日(星期二) 17:49
> 收件人:user <user@kylin.apache.org>; 旭鹤 <xuhe@dianjia.io>
> 主 题:Re:kylin help
>
> Hi,
> Currently you can set the configuration: "kylin.storage.partition.max-scan-bytes",
> to limit the scan bytes for each region to prevent possible OOM in region
> server, by default it is set to 3GB, and the configuration can be set in
> cube level.
> In our private Kylin env, we add two new configuration to limit the max
> return bytes and max return rows for each region to do more exact resource
> control, if community thinks that make sense, I can contribute the patch.
>
> 在 2018-04-24 16:22:50,"旭鹤" <xuhe@dianjia.io> 写道:
> 当并发量上来的时候  很多请求超时没响应。三台query 机中只有某一台出现问题,但是另外两台也无法查询
> 请问是不是协处理器的问题,协处理 有什么需要配置的么?
> 报错信息:
> kylin.log:Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: org.
> apache.hadoop.hbase.DoNotRetryIOException: Coprocessor passed deadline!
> Maybe server is overloaded
> kylin.log-      at org.apache.kylin.storage.hbase.cube.v2.
> coprocessor.endpoint.CubeVisitService.checkDeadline(
> CubeVisitService.java:225)
> kylin.log-      at org.apache.kylin.storage.hbase.cube.v2.
> coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
> kylin.log-      at org.apache.kylin.storage.hbase.cube.v2.
> coprocessor.endpoint.generated.CubeVisitProtos$
> CubeVisitService.callMethod(CubeVisitProtos.java:5555)
> kylin.log-      at org.apache.hadoop.hbase.regionserver.
> HRegion.execService(HRegion.java:7931)
> kylin.log-      at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> execServiceOnRegion(RSRpcServices.java:1969)
> kylin.log-      at org.apache.hadoop.hbase.regionserver.
> RSRpcServices.execService(RSRpcServices.java:1951)
> kylin.log-      at org.apache.hadoop.hbase.protobuf.
> generated.ClientProtos$ClientService$2.callBlockingMethod(
> ClientProtos.java:33652)
> kylin.log-      at org.apache.hadoop.hbase.ipc.RpcServer.
> call(RpcServer.java:2191)
> kylin.log-      at org.apache.hadoop.hbase.ipc.CallRunner.
> run(CallRunner.java:112)
> kylin.log-      at org.apache.hadoop.hbase.ipc.RpcExecutor$
> Handler.run(RpcExecutor.java:183)
> --
> kylin.log-      at org.apache.hadoop.hbase.ipc.
> RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.
> java:107)
> kylin.log-      at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.
> callMethod(CoprocessorRpcChannel.java:56)
> kylin.log-      at org.apache.kylin.storage.hbase.cube.v2.
> coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.
> visitCube(CubeVisitProtos.java:5616)
> kylin.log-      at org.apache.kylin.storage.hbase.cube.v2.
> CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:237)
> kylin.log-      at org.apache.kylin.storage.hbase.cube.v2.
> CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:206)
> kylin.log-      at org.apache.hadoop.hbase.client.HTable$15.
> call(HTable.java:1800)
> kylin.log-      at java.util.concurrent.FutureTask.run(
> FutureTask.java:266)
> kylin.log-      at java.util.concurrent.ThreadPoolExecutor.
> runWorker(ThreadPoolExecutor.java:1149)
> kylin.log-      at java.util.concurrent.ThreadPoolExecutor$
> Worker.run(ThreadPoolExecutor.java:624)
> kylin.log-      ... 1 more
> kylin.log:Caused by: org.apache.hadoop.hbase.ipc.
> RemoteWithExtrasException(org.apache.hadoop.hbase.
> DoNotRetryIOException): org.apache.hadoop.hbase.DoNotRetryIOException:
> Coprocessor passed deadline! Maybe server is overloaded
> kylin.log-      at org.apache.kylin.storage.hbase.cube.v2.
> coprocessor.endpoint.CubeVisitService.checkDeadline(
> CubeVisitService.java:225)
> kylin.log-      at org.apache.kylin.storage.hbase.cube.v2.
> coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
> kylin.log-      at org.apache.kylin.storage.hbase.cube.v2.
> coprocessor.endpoint.generated.CubeVisitProtos$
> CubeVisitService.callMethod(CubeVisitProtos.java:5555)
> kylin.log-      at org.apache.hadoop.hbase.regionserver.
> HRegion.execService(HRegion.java:7931)
> kylin.log-      at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> execServiceOnRegion(RSRpcServices.java:1969)
> kylin.log-      at org.apache.hadoop.hbase.regionserver.
> RSRpcServices.execService(RSRpcServices.java:1951)
> kylin.log-      at org.apache.hadoop.hbas
>
>
>
>
>
>
>
>
>
>
>
>
>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message