hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anoop Sam John (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-20003) WALLess HBase on Persistent Memory
Date Thu, 15 Feb 2018 07:14:00 GMT
Anoop Sam John created HBASE-20003:

             Summary: WALLess HBase on Persistent Memory
                 Key: HBASE-20003
                 URL: https://issues.apache.org/jira/browse/HBASE-20003
             Project: HBase
          Issue Type: New Feature
            Reporter: Anoop Sam John
            Assignee: Anoop Sam John

This JIRA aims to make use of persistent memory (pmem) technologies in HBase. 
One such usage is to make the Memstore to reside on pmem. Making a persistent memstore would
remove the need for WAL and paves way for a WALLess HBase. 

The existing region replica feature could be used here and ensure the data written to memstores
are synchronously replicated to the replicas and ensure strong consistency of the data. (pipeline

Advantages :
-Data Availability : Since the data across replicas are consistent (synchronously written)
our data is always 100 % available.
-Lower MTTR : It becomes easier/faster to switch over to the replicas on a primary region
failure as there is no WAL replay involved. Building the memstore map data also is much faster
than reading the WAL and replaying the WAL.
-Possibility of bigger memstores : These pmems are designed to have more memory than DRAMs
so it would also enable us to have bigger sized memstores which leads to lesser flushes/compaction
-Removes the dependency of HDFS on the write path

Initial PoC has been designed and developed. Testing is underway and we would publish the
PoC results along with the design doc sooner. The PoC doc will talk about the design decisions,
the libraries considered to work with these pmem devices, pros and cons of those libraries
and the performance results.

Note : Next gen memory technologies using 3DXPoint gives persistent memory feature. Such memory
DIMMs are soon to appear in the market. The PoC is done around Intel's ApachePass (AEP)

This message was sent by Atlassian JIRA

View raw message