hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12848) Utilize Flash storage for WAL
Date Tue, 13 Jan 2015 21:32:35 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14275978#comment-14275978
] 

Sean Busbey commented on HBASE-12848:
-------------------------------------

+1 reorganizing to subtask. the WAL is the obvious first step towards HBase leveraging the
feature.

patch comments

{code}
diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java b/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
index eee5e83..ffc660b 100644
--- a/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
+++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
@@ -918,6 +918,14 @@ public final class HConstants {
   public static final String ENABLE_WAL_COMPRESSION =
     "hbase.regionserver.wal.enablecompression";
 
+  /** Configuration name of HLog storage policy */
+  public static final String WAL_STORAGE_POLICY = "hbase.wal.storage.policy";
+  public static final String DEFAULT_WAL_STORAGE_POLICY = "NONE";
+  /** place only one replica in SSD and the remaining in default storage */
+  public static final String WAL_STORAGE_POLICY_ONE_SSD = "ONE_SSD";
+  /** place all replica on SSD */
+  public static final String WAL_STORAGE_POLICY_ALL_SSD = "ALL_SSD";
+
{code}

please put these somewhere other than HConstants so that they can be scoped to LimitedPrivate(CONFIG)
instead of public. Do we want to flag them as unstable pending some benchmarks?

Maybe in DefaultWALProvider since it already talks about being FS-based? Also would be a good
place to add some javadocs about using it. we should include a note that we only allow setting
one policy for all FS-based WALs (and not e.g. one policy for meta and another for user-data).

Also, would be good to add NONE as a policy so that folks could expressly stick to not using
SSD for the wal. (agree that for now doing NONE as default is also good).

{code}
+      DistributedFileSystem dfs = (DistributedFileSystem)fs;
+      Class<? extends DistributedFileSystem> dfsClass = dfs.getClass();
+      Method m = null;
+      try {
+        m = dfsClass.getDeclaredMethod("setStoragePolicy",
+          new Class<?>[] { Path.class, String.class });
+        m.setAccessible(true);
+      } catch (NoSuchMethodException e) {
+        LOG.info("FileSystem doesn't support"
+            + " setStoragePolicy; --HDFS-7228 not available");
+      } catch (SecurityException e) {
+        LOG.info("Doesn't have access to setStoragePolicy on "
+            + "FileSystems --HDFS-7228 not available", e);
+        m = null; // could happen on setAccessible()
+      }
+      if (m != null) {
+        try {
+          m.invoke(dfs, this.fullPathLogDir, storagePolicy);
+          LOG.info("setting " + storagePolicy + " for " + this.fullPathLogDir);
+        } catch (Exception e) {
+          LOG.warn("Unable to set " + storagePolicy + " for " + this.fullPathLogDir, e);
+        }
+      }
{code}

Can we move the reflection into one of the filesystem utility classes so that other parts
of HBASE-6572 could make use of it later?

> Utilize Flash storage for WAL
> -----------------------------
>
>                 Key: HBASE-12848
>                 URL: https://issues.apache.org/jira/browse/HBASE-12848
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>         Attachments: 12848-v1.patch
>
>
> One way to improve data ingestion rate is to make use of Flash storage.
> HDFS is doing the heavy lifting - see HDFS-7228.
> We assume an environment where:
> 1. Some servers have a mix of flash, e.g. 2 flash drives and 4 traditional drives.
> 2. Some servers have all traditional storage.
> 3. RegionServers are deployed on both profiles within one HBase cluster.
> This JIRA allows WAL to be managed on flash in a mixed-profile environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message