kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanna" <prasann...@trinitymobility.com>
Subject If hbase region server is down,then will down or not?
Date Mon, 08 Jan 2018 05:39:55 GMT
Hi all,

I have small doubt regarding kylin ,If my hbase region server is down then
kylin service will down automatically or it will work. I am facing this
issue from longback.Sometimes in my cluster regionserver is getting down
then my kylin service is getting down.I am attaching the logs,can you please
confirm what is route cause.

 

Logs: kylin.log

----------------------

 

Thu Jan 04 14:27:44 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047253026, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475450, waitTime=11172, operationTimeout=10000 expired.

 

2018-01-04 11:58:03,998 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1019 : Opening socket connection to server
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:2181. Will not attempt to authenticate
using SASL (unknown error)

2018-01-04 11:58:03,999 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:864 : Socket connection established to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:2181, initiating session

2018-01-04 11:58:19,376 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS03.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from
server in 15377ms for sessionid 0x360bc7adc000011, closing socket connection
and attempting reconnect

2018-01-04 11:58:19,376 ERROR [pool-8-thread-1] dao.ExecutableDao:155 :
error get all Jobs:

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:

Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475451, waitTime=11282, operationTimeout=5000 expired.

 

                at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:147)

                at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture
.run(ResultBoundedCompletionService.java:65)

                at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)

                at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)

                at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: Call to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.

                at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1
261)

                at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)

                at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpc
Client.java:213)

                at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementati
on.callBlockingMethod(AbstractRpcClient.java:287)

                at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$Blocki
ngStub.scan(ClientProtos.java:32651)

                at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.j
ava:372)

                at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199
)

                at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)

                at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetry
ingCaller.java:200)

                at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:356)

                at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:330)

                at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:126)

                ... 4 more

Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.

                at
org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)

                at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)

                ... 14 more

2018-01-04 11:58:19,377 ERROR [pool-8-thread-1]
execution.ExecutableManager:269 : error get All Job Ids

org.apache.kylin.job.exception.PersistentException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:

Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475451, waitTime=11282, operationTimeout=5000 expired.

 

                at
org.apache.kylin.job.dao.ExecutableDao.getJobIds(ExecutableDao.java:156)

                at
org.apache.kylin.job.execution.ExecutableManager.getAllJobIds(ExecutableMana
ger.java:267)

                at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(Defa
ultScheduler.java:85)

                at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

                at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

                at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
301(ScheduledThreadPoolExecutor.java:180)

                at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch
eduledThreadPoolExecutor.java:294)

                at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)

                at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)

                at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=1, exceptions:

Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475451, waitTime=11282, operationTimeout=5000 expired.

 

                at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:147)

                at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture
.run(ResultBoundedCompletionService.java:65)

                ... 3 more

Caused by: java.io.IOException: Call to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.

                at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1
261)

                at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)

                at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpc
Client.java:213)

                at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementati
on.callBlockingMethod(AbstractRpcClient.java:287)

                at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$Blocki
ngStub.scan(ClientProtos.java:32651)

                at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.j
ava:372)

                at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199
)

                at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)

                at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetry
ingCaller.java:200)

                at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:356)

                at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:330)

                at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:126)

                ... 4 more

Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475451,
waitTime=11282, operationTimeout=5000 expired.

                at
org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)

                at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)

                ... 14 more

2018-01-04 11:58:19,377 WARN  [pool-8-thread-1]
threadpool.DefaultScheduler:127 : Job Fetcher caught a exception
java.lang.RuntimeException:
org.apache.kylin.job.exception.PersistentException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:

Thu Jan 04 14:28:03 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047272716, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475451, waitTime=11282, operationTimeout=5000 expired.

 

2018-01-04 11:58:31,002 INFO  [BadQueryDetector]
service.BadQueryDetector:160 : System free memory less than 100 MB. 0
queries running.

2018-01-04 11:58:39,176 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1019 : Opening socket connection to server
ICCC-THBDPS01.EYEWAY.local/10.82.0.17:2181. Will not attempt to authenticate
using SASL (unknown error)

2018-01-04 11:58:39,488 ERROR [pool-8-thread-1] dao.ExecutableDao:155 :
error get all Jobs:

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:

Thu Jan 04 14:28:39 GMT+08:00 2018,
RpcRetryingCaller{globalStartTime=1515047299715, pause=100, retries=1},
java.io.IOException: Call to ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException:
Call id=475452, waitTime=8173, operationTimeout=5000 expired.

 

                at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:147)

                at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture
.run(ResultBoundedCompletionService.java:65)

                at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)

                at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)

                at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: Call to
ICCC-THBDPS03.EYEWAY.local/10.82.0.19:16020 failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475452,
waitTime=8173, operationTimeout=5000 expired.

                at
org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1
261)

                at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1229)

                at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpc
Client.java:213)

                at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementati
on.callBlockingMethod(AbstractRpcClient.java:287)

                at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$Blocki
ngStub.scan(ClientProtos.java:32651)

                at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.j
ava:372)

                at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199
)

                at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)

                at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetry
ingCaller.java:200)

                at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:356)

                at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(
ScannerCallableWithReplicas.java:330)

                at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:126)

                ... 4 more

Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=475452,
waitTime=8173, operationTimeout=5000 expired.

                at
org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)

                at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1203)

                ... 14 more

2018-01-04 11:59:49,777 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPSM01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:1140 : Client session timed out, have not heard from
server in 40161ms for sessionid 0x360bc7adc000012, closing socket connection
and attempting reconnect

2018-01-04 11:59:31,385 INFO  [BadQueryDetector]
service.BadQueryDetector:160 : System free memory less than 100 MB. 0
queries running.

2018-01-04 11:58:39,488 INFO
[localhost-startStop-1-SendThread(ICCC-THBDPS01.EYEWAY.local:2181)]
zookeeper.ClientCnxn:864 : Socket connection established to
ICCC-THBDPS01.EYEWAY.local/10.82.0.17:2181, initiating session


Mime
View raw message