hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Uneven write request to regions
Date Fri, 15 Nov 2013 21:34:09 GMT
bq. you must have your customerId, timestamp in the rowkey since you query
on it

Have you looked at this API in Scan ?

  public Scan setTimeRange(long minStamp, long maxStamp)


Cheers


On Fri, Nov 15, 2013 at 1:28 PM, Asaf Mesika <asaf.mesika@gmail.com> wrote:

> The problem is that I do know my rowkey design, and it follows people's
> best practice, but generates a really bad use case which I can't seem to
> know how to solve yet.
>
> The rowkey as I said earlier is:
> <customerId><bucket><timestampInMs><uniqueId>
> So when ,for example, you have 1000 customer, and bucket ranges from 1 to
> 16, you eventually end up with:
> * 30k regions - What happens, as I presume: you start with one region
> hosting ALL customers, which is just one. As you pour in more customers and
> more data, the region splitting kicks in. So, after a while, you get to a
> situation in which most regions hosts a specific customerId, bucket and
> time duration. For example: customer #10001, bucket 6, 01/07/2013 00:00 -
> 02/07/2013 17:00.
> * Empty regions - the first really bad consequence of what I told before is
> that when the time duration is over, no data will ever be written to this
> region. and Worst - when the TTL you set (lets say 1 month) is over and
> it's 03/08/2013, this region gets empty!
>
> The thing is that you must have your customerId, timestamp in the rowkey
> since you query on it, but when you do, you will essentially get regions
> which will not get any more writes to them, and after TTL become zombie
> regions :)
>
> The second bad part of this rowkey design is that some customer will have
> significantly less traffic than other customers, thus in essence their
> regions will get written in a very slow rate compared with the high traffic
> customer. When this happens on the same RS - bam: the slow region Puts are
> causing the WAL Queue to get bigger over time, since its region never gets
> to Max Region Size (256MB in our case) thus never gets flushed, thus stays
> in the 1st WAL file. Until when? Until we hit max logs file permitted (32)
> and then regions are flushed forcely. When this happen, we get about 100
> regions with 3k-3mb store files. You can imagine what happens next.
>
> The weirdest thing here is that this rowkey design is very common - nothing
> fancy here, so in essence this phenomenon should have happened to a lot of
> people - but from some reason, I don't see that much writing about it.
>
> Thanks!
>
> Asaf
>
>
>
> On Fri, Nov 15, 2013 at 3:51 AM, Jia Wang <ramon@appannie.com> wrote:
>
> > Then the case is simple, as i said "check your row key design, you can
> find
> > the start and end row key for each region, from which you can know why
> your
> > request with a specific row key doesn't hit a specified region"
> >
> > Cheers
> > Ramon
> >
> >
> > On Thu, Nov 14, 2013 at 8:47 PM, Asaf Mesika <asaf.mesika@gmail.com>
> > wrote:
> >
> > > It's from the same table.
> > > The thing is that some <customerId> simply have less data saved in
> HBase,
> > > while others have x50 (max) data.
> > > I'm trying to check how people designed their rowkey around it, or had
> > > other out-of-the-box solution for it.
> > >
> > >
> > >
> > > On Thu, Nov 14, 2013 at 12:06 PM, Jia Wang <ramon@appannie.com> wrote:
> > >
> > > > Hi
> > > >
> > > > Are the regions from the same table? If it was, check your row key
> > > design,
> > > > you can find the start and end row key for each region, from which
> you
> > > can
> > > > know why your request with a specific row key doesn't hit a specified
> > > > region.
> > > >
> > > > If the regions are for different table, you may consider to combine
> > some
> > > > cold regions for some tables.
> > > >
> > > > Thanks
> > > > Ramon
> > > >
> > > >
> > > > On Thu, Nov 14, 2013 at 4:59 PM, Asaf Mesika <asaf.mesika@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Have anyone ran into a case where a Region Server is hosting
> regions,
> > > in
> > > > > which some regions are getting lots of write requests, and the rest
> > > gets
> > > > > maye 1/1000 of the rate of write requests?
> > > > >
> > > > > This leads to a situation where the HLog queue reaches its maxlogs
> > > limit
> > > > > since, those HLogs containing the puts from slow-write regions are
> > > > "stuck"
> > > > > until the region will flush. Since those regions barely make it to
> > > their
> > > > > 256MB flush limit (our configuration), they won't flush. The HLogs
> > > queue
> > > > > gets bigger due to the fast-write regions, until reaches the stress
> > > mode
> > > > of
> > > > > "We have too many logs".
> > > > > This in turn flushes out lots of regions, many of them (about 100)
> > are
> > > > > ultra small (10k - 3mb). After 3 rounds like this, the compaction
> > queue
> > > > > gets very big....in the end the region server drops dead, and this
> > load
> > > > > somehow is moved to another RS, ...
> > > > >
> > > > > We are running 0.94.7 with 30 RS.
> > > > >
> > > > > I was wondering how did people handled a mix of slow-write-rate and
> > > > > high-write-rate of regions in 1 RS? I was thinking of writing a
> > > customer
> > > > > load balancer, which keeps tabs on the write request count and
> > memstore
> > > > > size, and move all the slow-write regions to 20% of cluster RS
> > > dedicated
> > > > to
> > > > > slow regions, thus releasing the fast write regions to work freely.
> > > > >
> > > > > Since this issue is hammering our production, we're about to try
to
> > > > > shut-down the WAL, and risk losing some information in those
> > slow-write
> > > > > regions until we can come up with a better solution.
> > > > >
> > > > > Any advice would be highly appreciated.
> > > > >
> > > > > Oh - our rowkey is quite normal:
> > > > > <customerId><bucket><Timestamp><uniqueId>
> > > > >
> > > > > Thanks!
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message