kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "朱真龙" <183920...@qq.com>
Subject Re: If hbase region server is down,then will down or not?
Date Mon, 08 Jan 2018 05:52:06 GMT
hi,Prasanna,


     maybe it is beause few free memory left in your system.  I find this on your log :


   






------------------ Original ------------------
From:  "Prasanna";<prasanna.p@trinitymobility.com>;
Date:  Mon, Jan 8, 2018 02:09 PM
To:  "user"<user@kylin.apache.org>;

Subject:  If hbase region server is down,then will down or not?



  
Hi all,
 
I have small doubt regarding kylin ,If my hbase region server is down then kylin service will
down automatically or it will work. I am facing this issue from longback.Sometimes in my cluster
regionserver is getting down then my kylin service is getting down.I am attaching the logs,can
you please confirm what is route cause.
 
 
 
Logs: kylin.log
 
----------------------
 
 
 
Thu Jan 04 14:27:44 GMT+08:00 2018, RpcRetryingCaller{globalStartTime=1515047253026, pause=100,
retries=1}, java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475450, waitTime=11172,
operationTimeout=10000 expired.
 
 
 
2018-01-04 11:58:03,998 INFO  [localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1019 : Opening socket connection to server ICCC-THBDPS03.EYEWAY.local/10.82.0.19:2181.
Will not attempt to authenticate using SASL (unknown error)
 
2018-01-04 11:58:03,999 INFO  [localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:864 : Socket connection established to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:2181,
initiating session
 
2018-01-04 11:58:19,376 INFO  [localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from server in 15377ms
for sessionid 0x360bc7adc000011, closing socket connection and attempting reconnect
 
2018-01-04 11:58:19,376 ERROR [pool-8-thread-1] dao.ExecutableDao:155 : error get all Jobs:
 
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
 
Thu Jan 04 14:28:03 GMT+08:00 2018, RpcRetryingCaller{globalStartTime=1515047272716, pause=100,
retries=1}, java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451, waitTime=11282,
operationTimeout=5000 expired.
 
 
 
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
 
                at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
 
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 
                at java.lang.Thread.run(Thread.java:745)
 
Caused by: java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451, waitTime=11282,
operationTimeout=5000 expired.
 
                at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1261)
 
                at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
 
                at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
 
                at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
 
                at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
 
                at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:372)
 
                at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
 
                at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
 
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
 
                at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:356)
 
                at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:330)
 
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
 
                ... 4 more
 
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451, waitTime=11282,
operationTimeout=5000 expired.
 
                at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
 
                at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
 
                ... 14 more
 
2018-01-04 11:58:19,377 ERROR [pool-8-thread-1] execution.ExecutableManager:269 : error get
All Job Ids
 
org.apache.kylin.job.exception.PersistentException: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed after attempts=1, exceptions:
 
Thu Jan 04 14:28:03 GMT+08:00 2018, RpcRetryingCaller{globalStartTime=1515047272716, pause=100,
retries=1}, java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451, waitTime=11282,
operationTimeout=5000 expired.
 
 
 
                at org.apache.kylin.job.dao.ExecutableDao.getJobIds(ExecutableDao.java:156)
 
                at org.apache.kylin.job.execution.ExecutableManager.getAllJobIds(ExecutableManager.java:267)
 
                at org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:85)
 
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 
                at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 
                at java.lang.Thread.run(Thread.java:745)
 
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1,
exceptions:
 
Thu Jan 04 14:28:03 GMT+08:00 2018, RpcRetryingCaller{globalStartTime=1515047272716, pause=100,
retries=1}, java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451, waitTime=11282,
operationTimeout=5000 expired.
 
 
 
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
 
                at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
 
                ... 3 more
 
Caused by: java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451, waitTime=11282,
operationTimeout=5000 expired.
 
                at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1261)
 
                at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
 
                at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
 
                at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
 
                at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
 
                at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:372)
 
                at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
 
                at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
 
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
 
                at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:356)
 
                at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:330)
 
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
 
                ... 4 more
 
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451, waitTime=11282,
operationTimeout=5000 expired.
 
                at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
 
                at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
 
                ... 14 more
 
2018-01-04 11:58:19,377 WARN  [pool-8-thread-1] threadpool.DefaultScheduler:127 : Job Fetcher
caught a exception java.lang.RuntimeException: org.apache.kylin.job.exception.PersistentException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
 
Thu Jan 04 14:28:03 GMT+08:00 2018, RpcRetryingCaller{globalStartTime=1515047272716, pause=100,
retries=1}, java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451, waitTime=11282,
operationTimeout=5000 expired.
 
 
 
2018-01-04 11:58:31,002 INFO  [BadQueryDetector] service.BadQueryDetector:160 : System free
memory less than 100 MB. 0 queries running.
 
2018-01-04 11:58:39,176 INFO  [localhost-startStop-1-SendThread(ICCC-THBDPS01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1019 : Opening socket connection to server ICCC-THBDPS01.EYEWAY.local/10.82.0.17:2181.
Will not attempt to authenticate using SASL (unknown error)
 
2018-01-04 11:58:39,488 ERROR [pool-8-thread-1] dao.ExecutableDao:155 : error get all Jobs:
 
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
 
Thu Jan 04 14:28:39 GMT+08:00 2018, RpcRetryingCaller{globalStartTime=1515047299715, pause=100,
retries=1}, java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475452, waitTime=8173,
operationTimeout=5000 expired.
 
 
 
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
 
                at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
 
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 
                at java.lang.Thread.run(Thread.java:745)
 
Caused by: java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed
on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475452, waitTime=8173,
operationTimeout=5000 expired.
 
                at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1261)
 
                at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)
 
                at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
 
                at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
 
                at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
 
                at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:372)
 
                at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
 
                at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
 
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
 
                at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:356)
 
                at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:330)
 
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
 
                ... 4 more
 
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475452, waitTime=8173,
operationTimeout=5000 expired.
 
                at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
 
                at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)
 
                ... 14 more
 
2018-01-04 11:59:49,777 INFO  [localhost-startStop-1-SendThread(ICCC-THBDPSM01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from server in 40161ms
for sessionid 0x360bc7adc000012, closing socket connection and attempting reconnect
 
2018-01-04 11:59:31,385 INFO  [BadQueryDetector] service.BadQueryDetector:160 : System free
memory less than 100 MB. 0 queries running.
 
2018-01-04 11:58:39,488 INFO  [localhost-startStop-1-SendThread(ICCC-THBDPS01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:864 : Socket connection established to ICCC-THBDPS01.EYEWAY.local/10.82.0.17:2181,
initiating session
Mime
View raw message