hbase-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From li...@apache.org
Subject svn commit: r1462832 - /hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
Date Sat, 30 Mar 2013 18:19:04 GMT
Author: liyin
Date: Sat Mar 30 18:19:04 2013
New Revision: 1462832

URL: http://svn.apache.org/r1462832
Log:
[HBASE-8223] Fixes race condition between append and close in HLog

Author: rshroff

Summary:
The RegionServer was performing a shutdown as it got
YouAreDeadException. It closed the HLog and was waiting to close all the
regions where it got stuck.

The reason region close is stuck because it's trying to obtain a lock
which is occupied by the put op, which in turn is waiting for append to
HLog to complete.

There is race condition here is between append/sync and HLog.close()
work flow.

Scenario:

Thread 1 => doing the append
Thread 2 => doing HregionServer shutdown

Timeline:

t1> 1: Verifies that LogSyncer is not shutting down and HLog is not
closed and calls sync()
t2> 2: HRegionServer issued HLog.close()
t3> 2: In HLog.close(), it joins the LogSyncer thread, which signals all
the threads waiting on syncDone and exits.
t4> 1: In sync, it sees that the sync has not complete until its txd,
hence adds itself to the syncDone.await queue.

Note: at t4, it does not check whether the LogSyncer Thread is alive or
not, which caused this hang.

The fix is to check whether the LogSyncer thread is closing or not
before going to await for the sync to finish.

Test Plan: mr

Reviewers: liyintang, aaiyer

Reviewed By: aaiyer

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D756413

Task ID: 2222578

Modified:
    hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java

Modified: hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
URL: http://svn.apache.org/viewvc/hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java?rev=1462832&r1=1462831&r2=1462832&view=diff
==============================================================================
--- hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
(original)
+++ hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
Sat Mar 30 18:19:04 2013
@@ -1178,8 +1178,9 @@ public class HLog implements Syncable {
     /**
      * This method first signals the thread that there's a sync needed
      * and then waits for it to happen before returning.
+     * @throws IOException
      */
-    public void addToSyncQueue(boolean force, long txid) {
+    public void addToSyncQueue(boolean force, long txid) throws IOException {
       while (true) {
         // Don't bother if somehow our append was already hflushed
         // Check this without even acquiring the lock, in the hope
@@ -1193,6 +1194,11 @@ public class HLog implements Syncable {
         }
         lock.lock();
         try {
+
+          if (syncerShuttingDown) {
+            throw (new IOException("LogSyncer closing. Aborting the sync"));
+          }
+
           // Wake the thread
           queueEmpty.signal();
 
@@ -1210,7 +1216,7 @@ public class HLog implements Syncable {
     }
   }
 
-  public void sync() {
+  public void sync() throws IOException {
     sync(false);
   }
 
@@ -1218,8 +1224,9 @@ public class HLog implements Syncable {
    * This method calls the LogSyncer in order to group commit the sync
    * with other threads.
    * @param force For catalog regions, force the sync to happen
+   * @throws IOException
    */
-  public void sync(boolean force) {
+  public void sync(boolean force) throws IOException {
     sync(force, this.unflushedEntries.get());
   }
 
@@ -1228,8 +1235,9 @@ public class HLog implements Syncable {
    *
    * @param txid
    *          The transaction id that this call is interested in.
+   * @throws IOException
    */
-  public void sync(boolean force, long txid) {
+  public void sync(boolean force, long txid) throws IOException {
     logSyncerThread.addToSyncQueue(force, txid);
   }
 



Mime
View raw message