hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 茂军王 (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HBASE-14180) Change timeout - SocketTimeoutException because of callTimeout
Date Fri, 28 Aug 2015 01:37:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717888#comment-14717888
] 

茂军王 commented on HBASE-14180:
-----------------------------

1   I meet the similar problem: https://issues.apache.org/jira/browse/HBASE-14330.

2   I change my regular expression to solve RPC Timeout problem.But I don`t know why.

3   Is this helpful for you ?

4   I look forward to your deep analysis for HBase Scanner  , I want to know what cause the
problem,How to cause.

> Change timeout - SocketTimeoutException because of callTimeout
> --------------------------------------------------------------
>
>                 Key: HBASE-14180
>                 URL: https://issues.apache.org/jira/browse/HBASE-14180
>             Project: HBase
>          Issue Type: Bug
>          Components: hbase, regionserver, rpc, Zookeeper
>    Affects Versions: 1.1.1
>         Environment: Hadoop with Ambari 2.1.0
> HBase 1.1.1.2.3
> HDFS 2.7.1.2.3
> Zookeeper 3.4.6.2.3
> Phoenix 4.4.0.2.3
>            Reporter: Adrià V.
>
> HBase keeps throwing a timeout exception I have tryed every configuration I could think
about to increase it.
> Partial stacktrace:
> {quote}
> Caused by: java.io.IOException: Call to hdp-w-1.c.dks-hadoop.internal/10.240.2.235:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001,
operationTimeout=60000 expired.
>         at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1242)
>         at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1210)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
>         at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
>         at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
>         at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
>         ... 4 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001,
operationTimeout=60000 expired.
>         at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
>         at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184)
>         ... 13 more
> {quote}
> I've tryed editing config files and also setting config in Ambari with the next keys
to increase the timeout with no success:
> - hbase.rpc.timeout
> - dfs.socket.timeout
> - dfs.client.socket-timeout
> - zookeeper.session.timeout
> Also the Phoenix properties, but I think it's mostly an HBase issue:
> - phoenix.query.timeoutMs
> - phoenix.query.keepAliveMs
> Full stack trace: 
> {quote}
> Error: Encountered exception in sub plan [0] execution. (state=,code=0)
> java.sql.SQLException: Encountered exception in sub plan [0] execution.
>         at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:157)
>         at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:251)
>         at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241)
>         at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>         at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240)
>         at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250)
>         at sqlline.Commands.execute(Commands.java:822)
>         at sqlline.Commands.sql(Commands.java:732)
>         at sqlline.SqlLine.dispatch(SqlLine.java:808)
>         at sqlline.SqlLine.begin(SqlLine.java:681)
>         at sqlline.SqlLine.start(SqlLine.java:398)
>         at sqlline.SqlLine.main(SqlLine.java:292)
> Caused by: org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException:
Failed after attempts=36, exceptions:
> Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000,
callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:542)
>         at org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
>         at org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
>         at org.apache.phoenix.join.HashCacheClient.serialize(HashCacheClient.java:106)
>         at org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:82)
>         at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:339)
>         at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:136)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:172)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException:
Failed after attempts=36, exceptions:
> Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000,
callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
>         at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:202)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:538)
>         ... 11 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
exceptions:
> Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000,
callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
>         at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:56)
>         at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:104)
>         at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)
>         at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)
>         at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)
>         at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:97)
>         at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:85)
>         ... 5 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36,
exceptions:
> Mon Aug 03 16:47:06 UTC 2015, null, java.net.SocketTimeoutException: callTimeout=60000,
callDuration=60303: row '' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
>         at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:223)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
>         at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
>         at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:403)
>         at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
>         at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:50)
>         ... 11 more
> Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row
'' on table 'hive_post_topics' at region=hive_post_topics,,1438084107396.cdbdc246ff0b7dfed31d481e0bccd2b5.,
hostname=hdp-w-1.c.dks-hadoop.internal,16020,1438619912282, seqNum=45322
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
>         at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
>         ... 3 more
> Caused by: java.io.IOException: Call to hdp-w-1.c.dks-hadoop.internal/10.240.2.235:16020
failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001,
operationTimeout=60000 expired.
>         at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1242)
>         at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1210)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
>         at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
>         at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
>         at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
>         ... 4 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=43, waitTime=60001,
operationTimeout=60000 expired.
>         at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
>         at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1184)
>         ... 13 more
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message