hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prakash Khemani (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook interferes with client's operations
Date Wed, 01 Sep 2010 23:34:53 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905313#action_12905313
] 

Prakash Khemani commented on HBASE-2952:
----------------------------------------

Yes, removing the HCM shutdown hook seems like the right thing to do. I don't understand the
TableOutputFormat part much ... so I am wondering, with the HCM shutdown hook gone, what will
the HBase user have to do to do a clean shutdown?







> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
>                 Key: HBASE-2952
>                 URL: https://issues.apache.org/jira/browse/HBASE-2952
>             Project: HBase
>          Issue Type: Improvement
>          Components: client
>    Affects Versions: 0.20.0
>            Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT
or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already
done one of the incrementColValue() then I want that thread to complete the other incrementColValue()
and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals'
all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down
all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems
with this - the incrementColValiue() didn't go through which will increase the chances of
inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages
that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed
connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903):
Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020,
location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO  [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206):
Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212):
<localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper
again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964):
Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r&
B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857):
locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client
is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470):
<localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489):
<localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server
got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116):
Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring
to it have been closed. That way the client can control when the shutdown actually happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message