hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imran M Yousuf <imyou...@gmail.com>
Subject Re: newbie: need help on understanding HBase
Date Fri, 13 Nov 2009 02:32:46 GMT
Thanks a lot :) ......

Imran

On Fri, Nov 13, 2009 at 9:19 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
> HBase is semi-column oriented. Column families is the storage model -
> everything in a column family is stored in a file linearly in HDFS.
> That means accessing data from a column family is really cheap and
> easy.  Adding more column families adds more files - it has the
> performance profile of adding new tables, except you dont actually
> have additional tables, so the conceptual complexity stays low.
>
> Data is stored at the "intersection" of the rowid, and the column
> family + qualifier.  This is sometimes called a "Cell" - contains a
> timestamp as well. You can have multiple versions all timestamped.
> The timestamp is by default the int64/java system milli time. I have
> to recommend against setting the timestamp explicitly if you can avoid
> it. So when you retrieve a row, you can get everything, a list of
> column qualifiers or a list of families or any combo.  (eg: list of
> these qualifiers out of family A and everything from family B)
>
> One problem with mysql is that large values tend to be pushed aside
> and require double block seeks to get them.  HBase can help you avoid
> this by giving you column families, eg: store the giant data in one
> family, smaller meta data in the second.
>
> The terms to use are:
> - Column family (or just family): the unit of locality in hbase.
> Everything in a family is stored in 1 (or a set) of files. A table is
> a name and a list of families with attributes for those families (eg:
> compression).  A family is a string.
> - Column qualifier (or just qualifier): allows you to store multiple
> values for the same row in 1 family. This value is a byte array and
> can be anything. The API converts null => new byte[0]{}.  This is the
> tricky bit, since most people dont think of "column names" as being
> dynamic.
> - Cell - the old name for a value + timestamp.  The new API (see:
> class Result) doesn't use this term, instead provides a different path
> to read data.
>
> You can use HBase as a normal datastore and use static names for the
> qualifiers, and that is just fine. But if you need something special
> to get past the lack of relations, you can start to do fun things with
> the qualifier as data. Building a secondary index for example.  The
> row key would be the secondary value (eg: city) and the qualifier
> would be the primary key (eg: userid) and the value would be a
> placeholder to indicate the value exists.
>
>
>
>
>
>
>
> On Thu, Nov 12, 2009 at 6:04 PM, Imran M Yousuf <imyousuf@gmail.com> wrote:
>> On Fri, Nov 13, 2009 at 9:00 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>>> HBase does at least 3 things that traditional databases have a hard time with:
>>>
>>> - Large blobs of data. Mysql is particularly guilty of not handling this well.
>>> - Tables that grow to be larger than reasonably priced single machines.
>>> - Write loads that are not compatible with master-slave replication
>>>
>>> The 2nd and 3rd are very interesting, since you either have to pay for
>>> something like Oracle RAC, or start sharding.
>>>
>>
>> Exactly, and since contents will be blob data and my experience with
>> RDBMS blob suggests that scaling is proportional to *BIG* money so I
>> am eager to take the HBase path. I was actually praying and hoping you
>> join this thread :). Can you please elaborate Column Family, Column
>> and Cell and their basic use cases?
>>
>> Thanks a lot,
>>
>> Imran
>>
>>> On Thu, Nov 12, 2009 at 5:58 PM, Imran M Yousuf <imyousuf@gmail.com> wrote:
>>>> On Thu, Nov 12, 2009 at 10:50 PM, Chris Bates
>>>> <christopher.andrew.bates@gmail.com> wrote:
>>>>> Hi Imran,
>>>>>
>>>>> I'm a new user as well.  I found these presentations helpful in answering
>>>>> most of your questions:
>>>>> http://wiki.apache.org/hadoop/HBase/HBasePresentations
>>>>>
>>>>> There are HBase schema designs in there.
>>>>>
>>>>
>>>> I read them, but without the speakers explanation the schema parts
>>>> remain unexplained for a dumb newbie like me. I was looking for more
>>>> concrete definitions of column family, column, cell etc. and their use
>>>> cases. I guess I will have to learn them by experimenting.
>>>>
>>>>> You might also want to read the original BigTable paper and the chapter
on
>>>>> HBase in OReilly's Hadoop book.
>>>>>
>>>>> But to answer one of your questions--"Big Data" usually refers to a dataset
>>>>> that is millions to billions in length.  But "Big Data" doesn't mean
you
>>>>> have to use a tool like HBase.  We have some MySQL tables that are 100
>>>>> million rows and work fine.  You have to identify what works best for
your
>>>>> use and use the most appropriate tool.
>>>>
>>>> Thanks, IMHO, I am sure that HBase is more suitable than MySQL simply
>>>> because of the complexity and cost in scaling an application with Blob
>>>> data.
>>>>
>>>> Thanks a lot,
>>>>
>>>> Imran
>>>>
>>>>>
>>>>> On Thu, Nov 12, 2009 at 9:13 AM, Imran M Yousuf <imyousuf@gmail.com>
wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I am absolutely new to HBase. All I have done is to read up
>>>>>> documentation, presentation and getting a single instance up and
>>>>>> running. I am starting on a Content Management System which will
be
>>>>>> used as a backend for multiple web applications of different natures.
>>>>>> In the CMS:
>>>>>> * User can define their content known as content type.
>>>>>> * Content can have  one-2-many one-2-one and many-2-many relationship
>>>>>> with other contents.
>>>>>> * Content fields should be versioned
>>>>>> * Content type can change in runtime, i.e. fields (a.k.a. columns
in
>>>>>> HBase) added and removal will not be allowed just yet.
>>>>>> * Every content type will have a corresponding grammer to validate
>>>>>> content of its type.
>>>>>> * It will have authentication and authorization
>>>>>> * It will have full text search based on Lucene/Katta.
>>>>>>
>>>>>> Based on these requirements I have the following questions that I
>>>>>> would like feedback on:
>>>>>> * Reading articles and presentations it looks to be HBase is a perfect
>>>>>> match as it supports multi-dimensional rows, versioned cells, dynamic
>>>>>> schema modification. But I could not understand what is the definition
>>>>>> of "Big Data" - that is if a content size is roughly 1~100kB
>>>>>> (field/cell size 0~100kB), is HBase meant for such uses?
>>>>>> * Since I am not sure how much load the site will have, I am planning
>>>>>> to setup DN+RS on Rackspace cloud instances with 2GB/80GB HDD with
a
>>>>>> view of with revenue and pageviews increasing, more moderate
>>>>>> "commodity" hardware can be added progressively. Any
>>>>>> comments/suggestions on this strategy?
>>>>>> * Where can I read up on or checkout samples RDBMS schemas converted
>>>>>> to HBase schema? Basically, I want to read up efficient schema design
>>>>>> for different cardinal relationships between objects.
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> --
>>>>>> Imran M Yousuf
>>>>>> Entrepreneur & Software Engineer
>>>>>> Smart IT Engineering
>>>>>> Dhaka, Bangladesh
>>>>>> Email: imran@smartitengineering.com
>>>>>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>>>>>> Mobile: +880-1711402557
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imran M Yousuf
>>>> Entrepreneur & Software Engineer
>>>> Smart IT Engineering
>>>> Dhaka, Bangladesh
>>>> Email: imran@smartitengineering.com
>>>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>>>> Mobile: +880-1711402557
>>>>
>>>
>>
>>
>>
>> --
>> Imran M Yousuf
>> Entrepreneur & Software Engineer
>> Smart IT Engineering
>> Dhaka, Bangladesh
>> Email: imran@smartitengineering.com
>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>> Mobile: +880-1711402557
>>
>



-- 
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557

Mime
View raw message