hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1024) Rearchitect regionserver I/O
Date Wed, 04 Mar 2009 07:41:56 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678622#action_12678622
] 

stack commented on HBASE-1024:
------------------------------

Bunch of us discussed the zero-copy above this evening at HUG6.  In particular remove HStoreKey
and just do byte array all the ways into the HFile and then on the way out, carry the HFile
key and value all the ways out to IPC.  Will make an issue once I have better handle on it.

> Rearchitect regionserver I/O
> ----------------------------
>
>                 Key: HBASE-1024
>                 URL: https://issues.apache.org/jira/browse/HBASE-1024
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: io, regionserver
>            Reporter: Andrew Purtell
>
> Consider supporting:
> * 2GB store files
> * 1TB per node (500 store files)
> * Cell values up to ~100MB
> * Typical use case of RS running with 1GB of heap only
> Some ideas:
> * Drop MapFile and make a custom store file format with (competing) design goals:
>       ** heap efficiency 
>       ** fast lookups
>       ** minimize I/O operations
>       ** optimize for typical DFS blocksizes (8MB, 64MB)
> * MRU cache for filehandles and store file indexes
> * Memory mapped store file indexes -- don't hold the indexes in heap; rely on the OS
blockcache for performance
> * "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message