hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Weishung Chung <weish...@gmail.com>
Subject Re: how to randomize the primary key which is a timestamp
Date Mon, 10 Jan 2011 17:42:32 GMT
Thanks alot, this will get me started :D

On Mon, Jan 10, 2011 at 11:04 AM, Matt Corgan <mcorgan@hotpads.com> wrote:

> You could have prefix = timestamp % 64.  Then for a single key lookup, you
> could calculate the prefix and query just one shard.  For a scan, you have
> to query all shards and merge the results.
>
>
> On Mon, Jan 10, 2011 at 11:56 AM, Weishung Chung <weishung@gmail.com>
> wrote:
>
> > Thank you for your prompt response. I am a bit confused about the prefix.
> > If i were to use prefix for the timestamp key, when come to query time,
> how
> > should i specify the row key to search for? How do I know which prefix
> was
> > used for a certain timestamp and needs to be append to the timestamp for
> > querying?
> >
> > On Mon, Jan 10, 2011 at 10:41 AM, Matt Corgan <mcorgan@hotpads.com>
> wrote:
> >
> > > You can put them all in the same table.  If you prefix the keys when
> > > written, use a prefix filter when querying.  I would choose a prefix
> > window
> > > that's about 4 times the number of nodes.
> > >
> > >
> > > On Mon, Jan 10, 2011 at 11:30 AM, Ted Dunning <tdunning@maprtech.com>
> > > wrote:
> > >
> > > > If multiple tables have the same key distribution and count, then
> they
> > > will
> > > > have similar split points for their regions, but the locations of the
> > > > regions will be randomized.
> > > >
> > > > I wouldn't worry about this until you see evidence it is a problem.
> > > >
> > > > On Mon, Jan 10, 2011 at 8:20 AM, Weishung Chung <weishung@gmail.com>
> > > > wrote:
> > > >
> > > > > Thank you for the replies.
> > > > > Most of the queries, (70%) will be for scanning a range of
> > consecutive
> > > > > times, with some single timestamp query (30%)
> > > > > But there are multiple tables with the same range of timestamps,
> will
> > > all
> > > > > these same range of timestamps from multiple tables be stored on
> the
> > > same
> > > > > region server and if so, could it affect the performance of map
> > reduce
> > > > jobs
> > > > > (operated on those tables with the same range of time periods) ?
> > Would
> > > > > hotspotting defeat the purpose of map reduce?
> > > > >
> > > > > On Mon, Jan 10, 2011 at 10:08 AM, Matt Corgan <mcorgan@hotpads.com
> >
> > > > wrote:
> > > > >
> > > > > > You can also add a random (or hashed) prefix to the beginning
of
> > the
> > > > key.
> > > > > >  If your prefix were one byte with values 0-63, you've divided
> the
> > > hot
> > > > > spot
> > > > > > into 64 smaller ones, which is better for writing.  The downside
> is
> > > > that
> > > > > if
> > > > > > you want to read a range of values, you will have to query all
64
> > > > > "shards"
> > > > > > and merge the sorted values.  You can choose whatever prefix
size
> > is
> > > > best
> > > > > > for your scenario.
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 10, 2011 at 11:05 AM, Chirstopher Tarnas <
> > cft@email.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Some options that I am aware of:
> > > > > > >
> > > > > > > reverse the byte order of the timestamp
> > > > > > > use UUIDs rather than a timestamp
> > > > > > > use hashing, this working really depends on your requirements
> > > > > > >
> > > > > > > On Mon, Jan 10, 2011 at 9:33 AM, Weishung Chung <
> > > weishung@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > What is the good way to randomize the primary key
which is a
> > > > > timestamp
> > > > > > in
> > > > > > > > HBase to avoid hotspotting?
> > > > > > > > Thank you so much :)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message