hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13960) HConnection stuck with UnknownHostException
Date Tue, 19 Jan 2016 15:04:39 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106845#comment-15106845
] 

Yu Li commented on HBASE-13960:
-------------------------------

We encountered the same issue months ago in our 0.98.12 cluster, and fixed it with a similar
solution. When trying to contribute our work back along with our upgrading to 1.x, found this
one by searching.

The patch here is a little stale that cannot compile with latest code base. After some minor
change, it works and the UT design is good enough to produce the issue. UT could pass w/ patch
(main changes on AbstractRpcClient), and fail with below exceptions w/o patch:

If UnknownHostException thrown during locate region in meta
{noformat}
2016-01-19 22:07:14,712 DEBUG [Time-limited test] client.ConnectionImplementation(944): locateRegionInMeta
parentTable=hbase:meta, metaLocation=, attempt=0 of 36 failed; retrying after sleep of 100
because: Failed after attempts=36, exceptions:
Tue Jan 19 22:07:14 CST 2016, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68243:
row 'testRegionServerRandomRpcFail,row,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
hostname=localhost,55113,1453212381028, seqNum=0
{noformat}

If thrown during puts:
{noformat}
2016-01-19 22:04:25,685 INFO  [hconnection-0x3eeadfbb-shared-pool14-t1] client.AsyncProcess$AsyncRequestFutureImpl(1222):
#12, table=testRegionServerRandomRpcFail, attempt=24/36 failed=1ops, last exception: java.net.UnknownHostException:
unknown host: random_invalid_host on localhost,54943,1453211970421, tracking started null,
retrying after=20005ms, replay=1ops
2016-01-19 22:04:25,695 INFO  [main] client.AsyncProcess(1711): #12, waiting for some tasks
to finish. Expected max=0, tasksInProgress=24
2016-01-19 22:04:45,698 INFO  [hconnection-0x3eeadfbb-shared-pool14-t1] client.AsyncProcess$AsyncRequestFutureImpl(1222):
#12, table=testRegionServerRandomRpcFail, attempt=25/36 failed=1ops, last exception: java.net.UnknownHostException:
unknown host: random_invalid_host on localhost,54943,1453211970421, tracking started null,
retrying after=20176ms, replay=1ops
2016-01-19 22:04:45,698 INFO  [main] client.AsyncProcess(1711): #12, waiting for some tasks
to finish. Expected max=0, tasksInProgress=25
{noformat}

Will upload the updated patch soon.

> HConnection stuck with UnknownHostException 
> --------------------------------------------
>
>                 Key: HBASE-13960
>                 URL: https://issues.apache.org/jira/browse/HBASE-13960
>             Project: HBase
>          Issue Type: Bug
>          Components: hbase
>    Affects Versions: 0.98.8
>            Reporter: Kurt Young
>         Attachments: HBASE-13960-0.98-v1.patch, HBASE-13960-v2.patch
>
>
> when put/get from hbase, if we meet a temporary dns failure causes resolve RS's host,
the error will never recovered. put/get will failed with UnknownHostException forever. 
> I checked the code, and the reason maybe:
> 1. when RegionServerCallable or MultiServerCallable prepare(), it gets a  ClientService.BlockingInterface
stub from Hconnection
> 2. In HConnectionImplementation::getClient, it caches the stub with a BlockingRpcChannelImplementation
> 3. In BlockingRpcChannelImplementation(), 
>      this.isa = new InetSocketAddress(sn.getHostname(), sn.getPort()); If we meet a 
temporary dns failure then the "address" in isa will be null.
> 4. then we launch the real rpc call, the following stack is:
> Caused by: java.net.UnknownHostException: unknown host: xxx.host2
> 	at org.apache.hadoop.hbase.ipc.RpcClient$Connection.<init>(RpcClient.java:385)
> 	at org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351)
> 	at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1523)
> 	at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1435)
> 	at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
> 	at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
> Besides, i noticed there is a protection in RpcClient:
> if (remoteId.getAddress().isUnresolved()) {
>         throw new UnknownHostException("unknown host: " + remoteId.getAddress().getHostName());
>       }
> shouldn't we do something when this situation occurred? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message