hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11339) HBase LOB
Date Tue, 17 Jun 2014 18:29:05 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034172#comment-14034172

Jonathan Hsieh commented on HBASE-11339:

>>We could avoid extra writes by just writing to a separate LOB log/file. Was this considered?
>It was considered. But we didn't find a good solution for this.
>  You mean to write the WAL by stores? If we use the HLog as the Lob files directly, is
it efficient to seek a KV in it? I don't think so.

I'm not convinced.  The idea I'm suggesting is having a special lob log file that is written
once at write time that is essentially the lob store files in the doc, and put a reference
to it (file name, and offset) in the normal wal.  This allows the lob to only be written once.
 I don't see how this would be less efficient than an approach that must write the values
out at least twice.

>>5MB cells are large but aren't really that big. Maybe this should just be "blobs"
(binary large objects) or "mobs" (medium objects)? the objects being immutable is important
>Actually the Lobs could be mutable. The Lobs that are not used anymore will be handled
by the Sweep Tool.

When I say mutable I mean that I can modify a particular byte in the lob without having to
"overwrite" the previous lob with a whole new lob.  I don't think the proposed design handles
this modify a few bytes in a large blob without doing the rewrite of the entire lob.

> Good idea. But In this way, all the actions occurs in the client, each client writes
a new file in HDFS. It's hard to control the file size which consequently leads to too many
small files in HDFS probably.

I agree about the hdfs small files problem but I think we need to properly define what a LOB
is and the scope of this effort.  (hence my suggestion of Medium Objects -- MOBS).  

Consider storing and shipping real large objects (say 100MB's or GB's).  Here hbase's api
is insufficient.  We'd want a streaming api for that or allow the client to go to the file
system directly (which may be a security concern for some users).  

Consider storing and shipping moderately sized objects (say 100k's to 10MB's).  HBase's API
is still sufficient, but we'd want to avoid the write amplification problem.  The proposed
design does this, but I think it could go further to avoid a 2x write amplification if we
handled it at the logging portion of the write path as opposed to the flushing part of the
of the write path.

I'm under the impression we are solving the latter case here.  Is that correct?

> HBase LOB
> ---------
>                 Key: HBASE-11339
>                 URL: https://issues.apache.org/jira/browse/HBASE-11339
>             Project: HBase
>          Issue Type: New Feature
>          Components: regionserver, Scanners
>            Reporter: Jingcheng Du
>            Assignee: Jingcheng Du
>         Attachments: HBase LOB Design.pdf
>   It's quite useful to save the massive binary data like images, documents into Apache
HBase. Unfortunately directly saving the binary LOB(large object) to HBase leads to a worse
performance since the frequent split and compaction.
>   In this design, the LOB data are stored in an more efficient way, which keeps a high
write/read performance and guarantees the data consistency in Apache HBase.

This message was sent by Atlassian JIRA

View raw message