Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of asaf.mesika@gmail.com
 designates 209.85.214.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAMUu0w-1tQdbH+_jh=bTa2-6c0oSw1Q61+QvTszQOc6a6dsS5A@mail.gmail.com>
References: 
 <CAAYV3ta9BqyD=ZwJM8HM8H_+N7=h1w642c2HCXOSOCC9KvgACg@mail.gmail.com>
	<CALte62wmimMG_F2cenv9+Np+VxGLC59fL7vLRhg7EhgmqZHFXA@mail.gmail.com>
	<CAMUu0w-1tQdbH+_jh=bTa2-6c0oSw1Q61+QvTszQOc6a6dsS5A@mail.gmail.com>
Date: Mon, 29 Apr 2013 08:46:27 +0300
Message-ID: 
 <CA+r7YvnTaN1-g2zwPp-Oyz__vN4ZaFxJhVHz+ycJGnbTTtZEaA@mail.gmail.com>
Subject: Re: Schema Design Question
From: Asaf Mesika <asaf.mesika@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7b33d904dfc33904db796728

--047d7b33d904dfc33904db796728
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I actually don't see the benefit of saving the data into HBase if all you
do is read per job id and purges it. Why not accumulate into HDFS per job
id and then dump the file? The way I see it, HBase is good for querying
parts of your data, even if it is only 10 rows. In your case your average
is 1 billion, so streaming it from hdfs seems faster .

On Saturday, April 27, 2013, Enis S=C3=B6ztutar wrote:

> Hi,
>
> Interesting use case. I think it depends on job many jobId's you expect t=
o
> have. If it is on the order of thousands, I would caution against going t=
he
> one table per jobid approach, since for every table, there is some master
> overhead, as well as file structures in hdfs. If jobId's are managable,
> going with separate tables makes sense if you want to efficiently delete
> all the data related to a job.
>
> Also pre-splitting will depend on expected number of jobIds / batchIds an=
d
> their ranges vs desired number of regions. You would want to keep number =
of
> regions hosted by a single region server in the low tens, thus, your spli=
ts
> can be across jobs or within jobs depending on cardinality. Can you share
> some more?
>
> Enis
>
>
> On Fri, Apr 26, 2013 at 2:34 PM, Ted Yu <yuzhihong@gmail.com<javascript:;=
>>
> wrote:
>
> > My understanding of your use case is that data for different jobIds wou=
ld
> > be continuously loaded into the underlying table(s).
> >
> > Looks like you can have one table per job. This way you drop the table
> > after map reduce is complete. In the single table approach, you would
> > delete many rows in the table which is not as fast as dropping the
> separate
> > table.
> >
> > Cheers
> >
> > On Sat, Apr 27, 2013 at 3:49 AM, Cameron Gandevia <cgandevia@gmail.com<=
javascript:;>
> > >wrote:
> >
> > > Hi
> > >
> > > I am new to HBase, I have been trying to POC an application and have =
a
> > > design questions.
> > >
> > > Currently we have a single table with the following key design
> > >
> > > jobId_batchId_bundleId_uniquefileId
> > >
> > > This is an offline processing system so data would be bulk loaded int=
o
> > > HBase via map/reduce jobs. We only need to support report generation
> > > queries using map/reduce over a batch (And possibly a single column
> > filter)
> > > with the batchId as the start/end scan key. Once we have finished
> > > processing a job we are free to remove the data from HBase.
> > >
> > > We have varied workloads so a job could be made up of 10 rows, 100,00=
0
> > rows
> > > or 1 billion rows with the average falling somewhere around 10 millio=
n
> > > rows.
> > >
> > > My question is related to pre-splitting. If we have a billion rows al=
l
> > with
> > > the same batchId (Our map/reduce scan key) my understanding is we
> should
> > > perform pre-splitting to create buckets hosted by different regions.
> If a
> > > jobs workload can be so varied would it make sense to have a single
> table
> > > containing all jobs? Or should we create 1 table per job and pre-spli=
t
> > the
> > > table for the given workload? If we had separate table we could drop
> them
> > > when no longer needed.
> > >
> > > If we didn't have a separate table per job how should we perform
> > splitting?
> > > Should we choose our largest possible workload and split for that? ev=
en
> > > though 90% of our jobs would fall in the lower bound in terms of row
> > count.
> > > Would we experience any issue purging jobs of varying sizes if
> everything
> > > was in a single table?
> > >
> > > any advice would be greatly appreciated.
> > >
> > > Thanks
> > >
> >
>

--047d7b33d904dfc33904db796728--