accumulo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From els...@apache.org
Subject [1/6] git commit: ACCUMULO-2842 Allow configuration of FSDataOutputStream method used to sync WALs.
Date Fri, 23 May 2014 20:08:14 GMT
Repository: accumulo
Updated Branches:
  refs/heads/1.5.2-SNAPSHOT 74acc30ca -> f654e7faa
  refs/heads/1.6.1-SNAPSHOT adaf14693 -> 2bf3ebb3d
  refs/heads/master 0164991f4 -> 20a466be3


ACCUMULO-2842 Allow configuration of FSDataOutputStream method used to sync WALs.

A critical piece of the bigtable design is using write-ahead logs to ensure
recovery after node failure. The append only file structure is meant to provide
a durable log of events so that the correct state can be reconstructed after
a failure when key-values have not been yet serialized to disk.

Hadoop 2 offers two levels of durability in regards to writes to files in HDFS. HFlush
ensures that all others clients who try to read the file you just wrote data to will
see that new data that was written. HSync will ensure that the data you wrote for that
file is actually sent to the underlying disk. As such, HFlush does not provide durability
when the node crashes without that data being flushed to the drive, while HSync does. HSync
does, however, increase the amount of time needed for sync/flush to complete (as hitting disk
consistently is slower than not).


Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/f654e7fa
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/f654e7fa
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/f654e7fa

Branch: refs/heads/1.5.2-SNAPSHOT
Commit: f654e7faadb9e94e2923087af334a707a4d2c81a
Parents: 74acc30
Author: Josh Elser <elserj@apache.org>
Authored: Fri May 23 14:40:37 2014 -0400
Committer: Josh Elser <elserj@apache.org>
Committed: Fri May 23 15:46:55 2014 -0400

----------------------------------------------------------------------
 .../java/org/apache/accumulo/core/conf/Property.java  |  3 +++
 .../accumulo/server/tabletserver/log/DfsLogger.java   | 14 +++++++++-----
 2 files changed, 12 insertions(+), 5 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/accumulo/blob/f654e7fa/core/src/main/java/org/apache/accumulo/core/conf/Property.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index 557ca1f..3d82046 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -204,6 +204,9 @@ public enum Property {
       "The number of threads for the distributed workq.  These threads are used for copying
failed bulk files."),
   TSERV_WAL_SYNC("tserver.wal.sync", "true", PropertyType.BOOLEAN,
       "Use the SYNC_BLOCK create flag to sync WAL writes to disk. Prevents problems recovering
from sudden system resets."),
+  TSERV_WAL_SYNC_METHOD("tserver.wal.sync.method", "hsync", PropertyType.STRING, "The method
to invoke when sync'ing WALs. HSync will provide " +
+      "resiliency in the face of unexpected power outages, at the cost of speed. If method
is not available, the legacy 'sync' method " +
+      "will be used to ensure backwards compatibility with older Hadoop versions"),
   
   // properties that are specific to logger server behavior
   LOGGER_PREFIX("logger.", null, PropertyType.PREFIX, "Properties in this category affect
the behavior of the write-ahead logger servers"),

http://git-wip-us.apache.org/repos/asf/accumulo/blob/f654e7fa/server/src/main/java/org/apache/accumulo/server/tabletserver/log/DfsLogger.java
----------------------------------------------------------------------
diff --git a/server/src/main/java/org/apache/accumulo/server/tabletserver/log/DfsLogger.java
b/server/src/main/java/org/apache/accumulo/server/tabletserver/log/DfsLogger.java
index 120c844..01c2448 100644
--- a/server/src/main/java/org/apache/accumulo/server/tabletserver/log/DfsLogger.java
+++ b/server/src/main/java/org/apache/accumulo/server/tabletserver/log/DfsLogger.java
@@ -264,13 +264,17 @@ public class DfsLogger {
       else
         logFile = fs.create(logPath, true, fs.getConf().getInt("io.file.buffer.size", 4096),
replication, blockSize);
 
+      String syncMethod = conf.getConfiguration().get(Property.TSERV_WAL_SYNC_METHOD);
       try {
-        // sync: send data to datanodes
-        sync = logFile.getClass().getMethod("sync");
         try {
-          // hsych: send data to datanodes and sync the data to disk
-          sync = logFile.getClass().getMethod("hsync");
-        } catch (NoSuchMethodException ex) {}
+          // hsync: send data to datanodes and sync the data to disk
+          sync = logFile.getClass().getMethod(syncMethod);
+        } catch (NoSuchMethodException ex) {
+          log.warn("Could not find configured " + syncMethod + " method, trying to fall back
to old Hadoop sync method", ex);
+
+          // sync: send data to datanodes
+          sync = logFile.getClass().getMethod("sync");
+        }
       } catch (Exception e) {
         throw new RuntimeException(e);
       }


Mime
View raw message