hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张铎 <palomino...@gmail.com>
Subject Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Date Thu, 21 May 2015 03:10:02 GMT
Is there any comparison between HBASE-11339 and HDFS-7240?
Is their 'Object' a super set of our 'Medium Object'?

2015-05-21 10:38 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:

> This is a useful feature, Jon.
>
> I went over the mega-patch and left some comments on review board.
>
> I noticed that hbck was not included in the patch. Neither did I find a
> sub-task of HBASE-11339 that covers hbck.
>
> Do you or Jingcheng plan to add MOB-aware capability for hbck ?
>
> Cheers
>
> On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jon@cloudera.com> wrote:
>
> > Hi folks,
> >
> > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O
> > and compaction path that allows individual moderately sized values
> > (10k-10MB) to be stored so that write amplification is reduced when
> > compared to the normal I/O path.   At a high level, it provides alternate
> > flush and compaction mechanisms that segregates large cells into a
> separate
> > area where they are not subject to potentially frequent compaction and
> > splits that can be encountered in the normal I/O path. A more detailed
> > design doc can be found on the hbase-11339 jira.
> >
> > Jingcheng Du has been working on the mob feature for a while and Anoop,
> Ram
> > and I have been shepherding him through the design revisions and
> > implementation of the feature in the hbase-11339 branch.[2]
> >
> > The branch we are proposing to merge into master is compatible with
> HBase's
> > core functionality including snapshots, replication, shell support,
> behaves
> > well with table alters, bulk loads and does not require external MR
> > processes. It has been documented, and subject to many integration test
> > runs  (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
> > Performance testing of the feature shows what can be a 2x-3x throughput
> > improvement for workloads that contain mobs. These results can be seen on
> > the hbase 2.0 panel discussion slides from hbasecon (once published).
> >
> > Recently there have been some hfile encryption related shortcomings that
> we
> > could address in branch or in master.
> >
> > Earlier iterations of the feature has been tested in production by users
> > that Jingcheng has been responsible for.  A version has also been
> deployed
> > at users I have been responsible for.  Some of the folks from Huawei
> > (ashutosh) have also been submitting the recent encryption bug reports
> > against the hbase-11339 branch so there is some evidence of usage by
> them.
> >
> > The four of us  (Jingcheng, Ram, Anoop and I) are satisfied with the
> > feature and feel it is a good time to call a merge vote.  Ive posted a
> > megapatch version for folks who want to peruse the code. [3]
> >
> > What do you all think?
> >
> > Thanks,
> > Jingcheng, Jon, Ram, and Anoop.
> >
> > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > [2] https://github.com/apache/hbase/tree/hbase-11339
> > [3] https://reviews.apache.org/r/34475/
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message