hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-1024) Rearchitect regionserver I/O
Date Sun, 23 Nov 2008 22:41:44 GMT
Rearchitect regionserver I/O

                 Key: HBASE-1024
                 URL: https://issues.apache.org/jira/browse/HBASE-1024
             Project: Hadoop HBase
          Issue Type: Improvement
          Components: io, regionserver
            Reporter: Andrew Purtell

Consider supporting:

* 2GB store files

* 1TB per node (500 store files)

* Cell values up to ~100MB

* Typical use case of RS running with 1GB of heap only

Some ideas:

* Drop MapFile and make a custom store file format with (competing) design goals:
      ** heap efficiency 
      ** fast lookups
      ** minimize I/O operations
      ** optimize for typical DFS blocksizes (8MB, 64MB)

* MRU cache for filehandles and store file indexes

* Memory mapped store file indexes -- don't hold the indexes in heap; rely on the OS blockcache
for performance

* "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message