hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karthikeyan Muthukumarasamy <mkarthik.h...@gmail.com>
Subject Re: Questions on Table design for time series data
Date Fri, 05 Oct 2012 04:53:30 GMT
Thanks Eugeny. We are currently running some experiments based on your
suggestions!

On Thu, Oct 4, 2012 at 2:20 AM, Eugeny Morozov <emorozov@griddynamics.com>wrote:

> I'd suggest to think about manual major compactions and splits. Using
> manual compactions and bulkload allows to split HFiles manually. Like if
> you would like to read last 3 months more often that all others data, then
> you could have three HFiles for each month and one HFile for whole other
> stuff. Using scan.setTimestamps would allow to filter out only those three
> HFiles, thus scan would be faster.
>
> Moreover if you have TTL about one month there is no need to run it
> everyday (as in auto mode). Especially, when using bulkloads you basically
> control the size of output HFiles by having input of particular size. Say,
> you give input for last two weeks and have one HFile per regions for last
> two weeks.
>
> Using this new feature known as Coprocessor, you could hook up to the
> compactSelection process and alter the compaction chosing HFiles you would
> like to process. That allow to combine particular HFiles.
>
> All of that allow to run major compaction just once-twice in month - major
> compaction over huge amount of data is a heavy operation - the rare, the
> better.
>
> Though without monitoring and measurement it looks like early optimization.
>
>
> On Wed, Oct 3, 2012 at 10:59 PM, Jacques <whshub@gmail.com> wrote:
>
> > We're all volunteers here so we don't always have the time to fully
> > understand and plan others' schemas.
> >
> > In general your questions seemed to be worried about a lot of things that
> > may or may not matter depending on the specifics of your implementation.
> >  Without knowing those specifics it is hard to be super definitive.  You
> > seem to be very worried about the cost of compactions and retention.  Is
> > that because you're having issues now?
> >
> > Short answers:
> >
> > q1: Unless you have a good reason for splitting up into two tables, I'd
> > keep as one.  Pros: Easier to understand/better matches intellectual
> > understanding/allows checkAndPuts across both families/data is colocated
> > (server, not disk) on retrieval if you want to work with both groups
> > simultaneously using get, MR, etc.  Con: There will be some extra
> > merge/flush activity if the two columns grow at substantially different
> > rates.
> >
> > q2: 365*1000 regions is problematic (if that is what you're suggesting).
> >  Even with HFilev2 and partially loaded multi-level indexes, there is
> still
> > quite a bit of overhead per region.  I pointed you at the Jive thing in
> > part since hashing that value as a bucket seems a lot more reasonable.
> >  Additional Random idea: if you know retention policy on insert and your
> > data is immutable post insertion, consider shifting the insert timestamp
> > and maintain a single ttl.  Would require more client side code but would
> > allow configurable ttls while utilizing existing HBase infrastructure.
> >
> > q3: Sounds like you're prematurely optimizing here.  Maybe others would
> > disagree.  I'd use ttl until you find that isn't performant enough.  The
> > tension between flexibility and speed is clear here.  I'd say you either
> > need to pick specific ttls and optimize for that scenario via region
> > pruning (e.g. separate tables for each ttl type) or you need to use a
> more
> > general approach that leverages the per value ttl and compaction
> > methodology.  There is enough operational work managing an HBase/HDFS
> > cluster without having to worry about specialized region management.
> >
> > Jacques
> >
> > On Wed, Oct 3, 2012 at 11:31 AM, Karthikeyan Muthukumarasamy <
> > mkarthik.here@gmail.com> wrote:
> >
> > > Hi Jacques,
> > > Thanks for the response!
> > > Yes, I have seen the video before. It suggets usage of TTL based
> > retention
> > > implementation. In their usecase, Jive has a fixed retention say 3
> months
> > > and so they can pre-create regions for so many buckets, their bucket id
> > is
> > > DAY_OF_YEAR%retention_in_days. But, in our usecase, the retention
> period
> > is
> > > configurable, so pre-creationg regions based on retention will not
> work.
> > > Thats why we went for MMDD based buckets which is immune to retention
> > > period changes.
> > > Now that you know that Ive gone through that video from Jive, I would
> > > request you to re-read my specific questions and share your
> suggestions.
> > > Thanks & Regards
> > > MK
> > >
> > >
> > >
> > > On Wed, Oct 3, 2012 at 11:51 PM, Jacques <whshub@gmail.com> wrote:
> > >
> > > > I would suggest you watch this video:
> > > >
> > > >
> > >
> >
> http://www.cloudera.com/resource/video-hbasecon-2012-real-performance-gains-with-real-time-data/
> > > >
> > > > The jive guys solved a lot of the problems you're talking about and
> > > discuss
> > > > it in that case study.
> > > >
> > > >
> > > >
> > > > On Wed, Oct 3, 2012 at 6:27 AM, Karthikeyan Muthukumarasamy <
> > > > mkarthik.here@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > > Our usecase is as follows:
> > > > > We have time series data continuously flowing into the system and
> has
> > > to
> > > > be
> > > > > stored in HBase.
> > > > > Subscriber Mobile Number (a.k.a MSISDN) is the primary identifier
> > based
> > > > on
> > > > > which data is stored and later retrieved.
> > > > > There are two sets of parameters that get stored in every record
in
> > > > HBase,
> > > > > lets call them group1 and group2. The number of records that would
> > have
> > > > > group1 parameters would be approx. 6 per day and the same for
> group2
> > > > > parameters is approx. 1 per 3 days (their cardinality is
> different).
> > > > >
> > > > > Typically, the retention policy for group1 parameters is 3 months
> and
> > > for
> > > > > group2 parameters is 1 year. The read-pattern is as follows: An
> > online
> > > > > query would ask for records matching an MSISDN for a given date
> > range,
> > > > and
> > > > > the system needs to respond with all available data (both from
> group1
> > > and
> > > > > group2) satifying the MSISDN and data range filters.
> > > > >
> > > > > Question1:
> > > > > Alternative1: Create a single table with G1 and G2 as two column
> > > > families.
> > > > > Alternative2: Create two tables one for each group
> > > > > Which is the better alternative and what are the pros and cons?
> > > > >
> > > > >
> > > > > Question2:
> > > > > To achieve max. distribution during write and reasonable complexity
> > > > during
> > > > > read, we decided on the following row key design:
> > > > > <last 3 digits of MSISDN>,<MMDD>,<full MSISDN>
> > > > > We will manually pre-split regions for the table based on the
> <last 3
> > > > > digits of MSISDN>,<MMDD> part of row key
> > > > > So there are 1000 (from 3 digits of MSISDN) * 365 (from MMDD)
> buckets
> > > > that
> > > > > would translate to as many regions
> > > > > In this case, when retention is configured as < 1 year, the design
> > > looks
> > > > > optimal
> > > > > When retention is configured > 1 year, one region might store
data
> > for
> > > > more
> > > > > than 1 day (feb 1 of 2012 and also feb 1 of 2013), which means more
> > > data
> > > > is
> > > > > to be handled by hbase during compactions and read.
> > > > > An alternative Key design, which does not have the above
> disadvantage
> > > is:
> > > > > <last 3 digits of MSISDN>,<YYYYMMDD>,<full MSISDN>
> > > > > this way, in one region, there will be only 1 days data at any
> point,
> > > > > regardless of retention
> > > > > What are other pros & cons of the two key designs?
> > > > >
> > > > > Question3:
> > > > > In our usecase, delete happens only based on retention policy,
> where
> > > one
> > > > > days full data has to be deleted when rention period is crossed
> (for
> > > eg,
> > > > if
> > > > > retention is 30 days, on Apr 1 all the data for Mar 1 is deleted)
> > > > > What is the most optimal way to implement this retention policy?
> > > > > Alternative 1: TTL for column famil is configured and we leave it
> to
> > > > HBase
> > > > > to delete data during major compaction, but we are not sure of the
> > cost
> > > > of
> > > > > this major compaction happening in all regions at same time
> > > > > Alternative 2: Through key design logic mentioned before, if we
> > ensure
> > > > data
> > > > > for one day goes into one set of regions, can we use HBase APIs
> like
> > > > > HFileArchiver to programatically archive and drop regions?
> > > > >
> > > > > Thanks & Regards
> > > > > MK
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Evgeny Morozov
> Developer Grid Dynamics
> Skype: morozov.evgeny
> www.griddynamics.com
> emorozov@griddynamics.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message