hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Trivial Update of "Hbase/NewFileFormat" by stack
Date Fri, 03 Oct 2008 19:49:44 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by stack:
http://wiki.apache.org/hadoop/Hbase/NewFileFormat

------------------------------------------------------------------------------
- This page is for discussion related to [https://issues.apache.org/jira/browse/HBASE-61 HBASE-61,
Create an HBase-specific MapFile implementation].  That issue, and its linked issues, has
a bunch of suggestions for how we might do a better persistence.  Other related issues include,
[https://issues.apache.org/jira/browse/HADOOP-3315 TFile], and [https://issues.apache.org/jira/browse/HBASE-647
HBASE-647, Remove the HStoreFile 'info' file (and index and bloomfilter if possible)].
+ This page is for discussion related to [https://issues.apache.org/jira/browse/HBASE-61 HBASE-61,
Create an HBase-specific MapFile implementation].  That issue, and its linked issues, has
a bunch of suggestions for how we might do a better persistence.  Most have been replicated
in the ''New Format'' section below.  Other related issues include, [https://issues.apache.org/jira/browse/HADOOP-3315
TFile], and [https://issues.apache.org/jira/browse/HBASE-647 HBASE-647, Remove the HStoreFile
'info' file (and index and bloomfilter if possible)].
  
  == Current Implementation ==
  
@@ -35, +35 @@

  If index included offset to every key, would be able to use it to figure if file had an
entry for the queried key and every index lookup would get us exact offset.  But such an index
would be too large to keep in memory (If values are small, file could have many entries. 
Files are usually about 64MB but can grow to an upper-bound of about 1G though this is configurable
and nothing to stop it being configured up from this).
  
  == New Format ==
-  * [https://issues.apache.org/jira/browse/HBASE-647 HBASE-647]: Have data, metadata, indices
and bloomfilters, etc., all rolled up in the one file.  Could do this with [https://issues.apache.org/jira/browse/HADOOP-3315
TFile].
+  * [https://issues.apache.org/jira/browse/HBASE-647 HBASE-647]: Have data, metadata, indices
and bloomfilters, etc., all rolled up in the one file.  Could do this with [https://issues.apache.org/jira/browse/HADOOP-3315
TFile].  SequenceFile allows addition of metadata but this facility is not exposed in MapFile.
 Could add to MapFile but SequenceFile metadata is stored in the head of the SequenceFile.
 Many metadata are known only after the flush: count-of-entries, bloomfilter, etc.
+  * [https://issues.apache.org/jira/browse/HBASE-519 Convert HStore to use only new interface
methods].  If an Interface, can try different implementations.
+  * In-memory: TFile has user supply the underlying data stream.  Could supply a stream hosted
in  memory.
+  * Always-on General bloomfilter. We know how many entries a file will have when we go to
flush it so we can optimally size a bloomfilter.  The small amount of memory a bloomfilter
occupies will pay for itself many-fold in the seeks saved trying to figure is a file contains
an asked for key.
+  * Optimal random-access
+  * Iterate over keys only, rather than mapfiles currenty key+values always.  This'd be useful
when trying to find closest. TFile and SequenceFile can do this (Its not exposed in MapFile).
+  
+ 
+ === Nice-to-haves ===
+  * Don't write out the family portion of column when writing keys.
  
  == Other File Formats ==
  Cassandra uses a Sequence File.  It adds key/values in blocks of 128 by default.  On the
128th entry, an index for the block keys is inlined and then a new block begins.  Block offsets
are kept out in an index file as in MapFile.  Bloomfilters are on by default.

Mime
View raw message