hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14431) AsyncRpcClient#removeConnection() never removes connection from connections pool if server fails
Date Sat, 19 Sep 2015 17:01:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14877211#comment-14877211
] 

Hudson commented on HBASE-14431:
--------------------------------

FAILURE: Integrated in HBase-TRUNK #6821 (See [https://builds.apache.org/job/HBase-TRUNK/6821/])
HBASE-14431 AsyncRpcClient#removeConnection() never removes connection from connections pool
if server fails (Samir Ahmic) (tedyu: rev 1545e1ed8d68b780dca49084cf5d8173481f72c0)
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcChannel.java


> AsyncRpcClient#removeConnection() never removes connection from connections pool if server
fails
> ------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-14431
>                 URL: https://issues.apache.org/jira/browse/HBASE-14431
>             Project: HBase
>          Issue Type: Bug
>          Components: IPC/RPC
>    Affects Versions: 2.0.0, 1.0.2, 1.1.2
>            Reporter: Samir Ahmic
>            Assignee: Samir Ahmic
>            Priority: Critical
>             Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
>         Attachments: HBASE-14431-v2.patch, HBASE-14431.patch
>
>
> I was playing with master branch in distributed mode (3 rs + master + backup_master)
and notice strange behavior when i was testing this sequence of events on single rs: /kill/start/run_balancer
while client was writing data to cluster (LoadTestTool).
> I have notice that LTT fails with following:
> {code}
> 2015-09-09 11:05:58,364 INFO  [main] client.AsyncProcess: #2, waiting for some tasks
to finish. Expected max=0, tasksInProgress=35
> Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: BindException: 1 time, 
> 	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> 	at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
> 	at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1697)
> 	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:211)
> {code}
> After some digging  and adding some more logging in code i have notice that following
condition in  {code}AsyncRpcClient.removeConnection(AsyncRpcChannel connection) {code} is
never true:
> {code}
> if (connectionInPool == connection) {
> {code} 
> causing that  {code}AsyncRpcChannel{code} connection is never removed from {code}connections{code}
pool in case rs fails.
> After changing this condition to:
> {code}
> if (connectionInPool.address.equals(connection.address)) {
> {code}
> issue was resolved and client was removing failed server from connections pool.
> I will attach patch after running some more tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message