hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: [jira] Commented: (HBASE-1197) IPC of large cells should transfer in chunks not via naive full copy
Date Thu, 12 Feb 2009 06:05:23 GMT
I doubt we could chunk values straight into HFile - you'd have to have 1
file for 1 value.  If your value is that large (more than hundreds of megs),
maybe you shouldn't be storing that in HBase - store directly in HDFS and
use HBase to index the content and provide filename pointers.

As it stands, right now a key/value has to live in memcache for some period
of time (seconds? minutes?), so storing an entire key/value has to be
feasable.  Not supporting chunking/streaming doesn't seem to be a major

I think of hbase as a way of making it possible to effiency store smallish
values on HDFS.  I think we should support reasonably large values, but
right now there is a 2gb value max size (int size for value).  With enough
RAM thrown at HBase it should be possible to support nearly all of that

On Wed, Feb 11, 2009 at 9:51 PM, stack (JIRA) <jira@apache.org> wrote:

>    [
> https://issues.apache.org/jira/browse/HBASE-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672889#action_12672889]
> stack commented on HBASE-1197:
> ------------------------------
> Its almost as though we should chunk end-to-end; from hfile all the ways on
> out to the client.
> Tough one is on the way in.  The big cell is going to have to sit in the
> mecache for some -- probably small if its a big cell value -- period of
> time.  Can we chunk on the way in?  From client to hfile?
> > IPC of large cells should transfer in chunks not via naive full copy
> > --------------------------------------------------------------------
> >
> >                 Key: HBASE-1197
> >                 URL: https://issues.apache.org/jira/browse/HBASE-1197
> >             Project: Hadoop HBase
> >          Issue Type: Improvement
> >            Reporter: Andrew Purtell
> >             Fix For: 0.20.0
> >
> >
> > Several instances of OOME when trying to serve up large cells to clients
> have been observed. IPC should send large cell content in chunks instead of
> as one large naive copy.
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message