Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of padmanaban.mathulu@gmail.com
 designates 74.125.82.169 as permitted sender)
MIME-Version: 1.0
Date: Thu, 26 Jul 2012 18:03:17 +0530
Message-ID: 
 <CAN5+yN8TWn-qQXXfj-FD=u4=P7cpShdgC1aiBAzfJuqOSZq8mw@mail.gmail.com>
Subject: HBase datamodel to purge old data
From: Padmanaban Mathulu <padmanaban.mathulu@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=f46d043bdb4ec84e1004c5bacc1e

--f46d043bdb4ec84e1004c5bacc1e
Content-Type: text/plain; charset=ISO-8859-1

Hi,

We have the following use case:

   -

   Store telecom CDR data on a per subscriber basis
   -

   data is time series based and every record is per-subscriber based
   -

   comes in round the clock
   -

   the expected volume of data would be around 300 million records/day.
   -

   this data is to be queried 24/7 by an online system where the filters
   are subscriber id and date range


 Since the volume of data is huge, we have data retention policies to
archive old data on a daily basis.
For example, if retention is set to 90 days, every day a offline process
would delete data from Hbase which is older than 90 days and archive it on
tape.


The current HBase data model design is as follows:

Separate table for every day's data with row key as subscriber id: reason
for this is bulk delete of one days data within a big table is more
expensive than dropping a one day table

In this per-day-separate-table model, the load balancer will never get
triggered as the current days table is always in memory, and daughter
regions will continuously get assigned to same region server. This leads to
a region server hotspots.

Please feedback on whether the per-day-separate-table model is the
best-practice for this use case considering the data life cycle management
requirement. If yes, how do we solve the side effect of region server
hotspot? If no, please advice alternate model.
 Thanks in advance
Padmanaban M

--f46d043bdb4ec84e1004c5bacc1e--