hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1024) Rearchitect regionserver I/O
Date Tue, 25 Nov 2008 00:37:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650413#action_12650413

stack commented on HBASE-1024:

Good stuff Andrew.  I concur.  Lets take on these goals.  Shall we put up a wiki page pointing
to this issue that talks of rearchitecting project?   and maybe belong in a working document
of their own.  Perhaps a page up on wiki -- like http://wiki.apache.org/hadoop/Hbase/NewFileFormat
-- with perhaps a link under roadmap or maybe we need an architectural goals section where
we stable these targets (and X-them out as we knock them off)?

> Rearchitect regionserver I/O
> ----------------------------
>                 Key: HBASE-1024
>                 URL: https://issues.apache.org/jira/browse/HBASE-1024
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: io, regionserver
>            Reporter: Andrew Purtell
> Consider supporting:
> * 2GB store files
> * 1TB per node (500 store files)
> * Cell values up to ~100MB
> * Typical use case of RS running with 1GB of heap only
> Some ideas:
> * Drop MapFile and make a custom store file format with (competing) design goals:
>       ** heap efficiency 
>       ** fast lookups
>       ** minimize I/O operations
>       ** optimize for typical DFS blocksizes (8MB, 64MB)
> * MRU cache for filehandles and store file indexes
> * Memory mapped store file indexes -- don't hold the indexes in heap; rely on the OS
blockcache for performance
> * "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message