hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qiang Tian <tian...@gmail.com>
Subject Re: Question on the number of column families
Date Thu, 07 Aug 2014 05:36:25 GMT
Hi TaeYun,
thanks for explain.




On Thu, Aug 7, 2014 at 12:50 PM, innowireless TaeYun Kim <
taeyun.kim@innowireless.co.kr> wrote:

> Hi Qiang,
> thank you for your help.
>
> 1. Regarding HBASE-5416, I think it's purpose is simple.
>
> "Avoid loading column families that is irrelevant to filtering while
> scanning."
> So, it can be applied to my 'dummy CF' case.
> That is, a dummy CF can act like an 'relevant' CF to filtering, provided
> that HBase can select it while applying a rowkey filter, since a dummy CF
> has the rowkey data in its 'dummy' KeyValue object.
>
> 2. About rowkey.
>
> What I meant is, I would include the field name as a component when the
> byte array for a rowkey is constructed.
>
> 3. About read-only-ness and the number of CF.
>
> Thank you for your suggestion.
> But since MemStore and BlockCache is separately managed on each column
> family, I'm a little concerned with the memory footprint.
>
> Thank you.
>
> -----Original Message-----
> From: Qiang Tian [mailto:tianq01@gmail.com]
> Sent: Thursday, August 07, 2014 11:43 AM
> To: user@hbase.apache.org
> Subject: Re: Question on the number of column families
>
> Hi,
> the description of hbase-5416 stated why it was introduced, if you only
> have 1 CF, dummy CF does not help. it is helpful for multi-CF case, e.g.
> "putting them in one column family. And "Non frequently" ones in another. "
>
> bq. "Field name will be included in rowkey."
> Please read the chapter 9 "Advanced usage" in book "HBase Definitive Guide"
> about how hbase store data on disk and how to design rowkey based on
> specific scenario.(rowkey is the only index you can use, so take care)
>
> bq. "The table is read-only. It is bulk-loaded once. When a new data is
> ready, A new table is created and the old table is deleted."
> the scenario is quite different.  as hbase is designed for random
> read/write.  the limitation described at
> http://hbase.apache.org/book/number.of.cfs.html is to consider the write
> case(flush&compaction), perhaps you could try 140 CFs, as long as you can
> presplit your regions well? after that,  since no write, there will be no
> flush/compaction...anyway, any idea better be tested with your real data.
>
>
>
>
>
>
>
>
> On Wed, Aug 6, 2014 at 7:00 PM, innowireless TaeYun Kim <
> taeyun.kim@innowireless.co.kr> wrote:
>
> > Hi Ted,
> >
> > Now I finished reading the filtering section and the source code of
> > TestJoinedScanners(0.94).
> >
> > Facts learned:
> >
> > - While scanning, an entire row will be read even for a rowkey filtering.
> > (Since a rowkey is not a physically separate entity and stored in
> > KeyValue object, it's natural. Am I right?)
> > - The key API for the essential column family support is
> > setLoadColumnFamiliesOnDemand().
> >
> > So, now I have questions:
> >
> > On rowkey filtering, which column family's KeyValue object is read?
> > If HBase just reads a KeyValue from a randomly selected (or just the
> > first) column family, how is setLoadColumnFamiliesOnDemand() affected?
> > Can HBase select a smaller column family intelligently?
> >
> > If setLoadColumnFamiliesOnDemand() can be applied to a rowkey
> > filtering, a 'dummy' column family can be used to minimize the scan cost.
> >
> > Thank you.
> >
> >
> > -----Original Message-----
> > From: innowireless TaeYun Kim [mailto:taeyun.kim@innowireless.co.kr]
> > Sent: Wednesday, August 06, 2014 1:48 PM
> > To: user@hbase.apache.org
> > Subject: RE: Question on the number of column families
> >
> > Thank you.
> >
> > The 'dummy' column will always hold the value '1' (or even an empty
> > string), that only signifies that this row exists. (And the real value
> > is in the other 'big' column family) The value is irrelevant since
> > with current schema the filtering will be done by rowkey components
> > alone. No column value is needed. (I will begin reading the filtering
> > section shortly
> > - it is only 6 pages ahead. So sorry for my premature thoughts)
> >
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > Sent: Wednesday, August 06, 2014 1:38 PM
> > To: user@hbase.apache.org
> > Subject: Re: Question on the number of column families
> >
> > bq. add a 'dummy' column family and apply HBASE-5416 technique
> >
> > Adding dummy column family is not the way to utilize essential column
> > family support - what would this dummy column family hold ?
> >
> > bq. since I have not read the filtering section of the book I'm
> > reading yet
> >
> > Once you finish reading, you can look at the unit test
> > (TestJoinedScanners) from HBASE-5416. You would understand this
> > feature better.
> >
> > Cheers
> >
> >
> > On Tue, Aug 5, 2014 at 9:21 PM, innowireless TaeYun Kim <
> > taeyun.kim@innowireless.co.kr> wrote:
> >
> > > Thank you all.
> > >
> > > Facts learned:
> > >
> > > - Having 130 column families is too much. Don't do that.
> > > - While scanning, an entire row will be read for filtering, unless
> > > HBASE-5416 technique is applied which makes only relevant column
> > > family is loaded. (But it seems that still one can't load just a
> > > column needed while
> > > scanning)
> > > - Big row size is maybe not good.
> > >
> > > Currently it seems appropriate to follow the one-column solution
> > > that Alok Singh suggested, in part since currently there is no
> > > reasonable grouping of the fields.
> > >
> > > Here is my current thinking:
> > >
> > > - One column family, one column. Field name will be included in rowkey.
> > > - Eliminate filtering altogether (in most case) by properly ordering
> > > rowkey components.
> > > - If a filtering is absolutely needed, add a 'dummy' column family
> > > and apply HBASE-5416 technique to minimize disk read, since the
> > > field value can be large(~5MB). (This dummy column thing may not be
> > > right, I'm not sure, since I have not read the filtering section of
> > > the book I'm reading yet)
> > >
> > > Hope that I am not missing or misunderstanding something...
> > > (I'm a total newbie. I've started to read a HBase book since last
> > > week...)
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message