hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: Millions of photos into Hbase
Date Tue, 21 Sep 2010 03:30:18 GMT
you need 2 more things:

- restart hdfs
- make sure the hadoop jar from your install replaces the one we ship with

On Mon, Sep 20, 2010 at 8:22 PM, Jack Levin <magnito@gmail.com> wrote:
> So, I switched to 0.89, and we already had CDH3
> (hadoop-0.20-datanode-0.20.2+320-3.noarch), even though I added
>  <name>dfs.support.append</name> as true to both hdfs-site.xml and
> hbase-site.xml, the master still reports this:
>  You are currently running the HMaster without HDFS append support
> enabled. This may result in data loss. Please see the HBase wiki  for
> details.
> Master Attributes
> Attribute Name  Value   Description
> HBase Version   0.89.20100726, r979826  HBase version and svn revision
> HBase Compiled  Sat Jul 31 02:01:58 PDT 2010, stack     When HBase version
> was compiled and by whom
> Hadoop Version  0.20.2, r911707 Hadoop version and svn revision
> Hadoop Compiled Fri Feb 19 08:07:34 UTC 2010, chrisdo   When Hadoop
> version was compiled and by whom
> HBase Root Directory    hdfs://namenode-rd.imageshack.us:9000/hbase     Location
> of HBase home directory
> Any ideas whats wrong?
> -Jack
> On Mon, Sep 20, 2010 at 5:47 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>> Hey,
>> There is actually only 1 active branch of hbase, that being the 0.89
>> release, which is based on 'trunk'.  We have snapshotted a series of
>> 0.89 "developer releases" in hopes that people would try them our and
>> start thinking about the next major version.  One of these is what SU
>> is running prod on.
>> At this point tracking 0.89 and which ones are the 'best' peach sets
>> to run is a bit of a contact sport, but if you are serious about not
>> losing data it is worthwhile.  SU is based on the most recent DR with
>> a few minor patches of our own concoction brought in.  If current
>> works, but some Master ops are slow, and there are a few patches on
>> top of that.  I'll poke about and see if its possible to publish to a
>> github branch or something.
>> -ryan
>> On Mon, Sep 20, 2010 at 5:16 PM, Jack Levin <magnito@gmail.com> wrote:
>>> Sounds, good, only reason I ask is because of this:
>>> There are currently two active branches of HBase:
>>>    * 0.20 - the current stable release series, being maintained with
>>> patches for bug fixes only. This release series does not support HDFS
>>> durability - edits may be lost in the case of node failure.
>>>    * 0.89 - a development release series with active feature and
>>> stability development, not currently recommended for production use.
>>> This release does support HDFS durability - cases in which edits are
>>> lost are considered serious bugs.
>>> Are we talking about data loss in case of datanode going down while
>>> being written to, or RegionServer going down?
>>> -jack
>>> On Mon, Sep 20, 2010 at 4:09 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>>>> We run 0.89 in production @ Stumbleupon.  We also employ 3 committers...
>>>> As for safety, you have no choice but to run 0.89.  If you run a 0.20
>>>> release you will lose data.  you must be on 0.89 and
>>>> CDH3/append-branch to achieve data durability, and there really is no
>>>> argument around it.  If you are doing your tests with 0.20.6 now, I'd
>>>> stop and rebase those tests onto the latest DR announced on the list.
>>>> -ryan
>>>> On Mon, Sep 20, 2010 at 3:17 PM, Jack Levin <magnito@gmail.com> wrote:
>>>>> Hi Stack, see inline:
>>>>> On Mon, Sep 20, 2010 at 2:42 PM, Stack <stack@duboce.net> wrote:
>>>>>> Hey Jack:
>>>>>> Thanks for writing.
>>>>>> See below for some comments.
>>>>>> On Mon, Sep 20, 2010 at 11:00 AM, Jack Levin <magnito@gmail.com>
>>>>>>> Image-Shack gets close to two million image uploads per day,
which are
>>>>>>> usually stored on regular servers (we have about 700), as regular
>>>>>>> files, and each server has its own host name, such as (img55).
>>>>>>> been researching on how to improve our backend design in terms
of data
>>>>>>> safety and stumped onto the Hbase project.
>>>>>> Any other requirements other than data safety? (latency, etc).
>>>>> Latency is the second requirement.  We have some services that are
>>>>> very short tail, and can produce 95% cache hit rate, so I assume this
>>>>> would really put cache into good use.  Some other services however,
>>>>> have about 25% cache hit ratio, in which case the latency should be
>>>>> 'adequate', e.g. if its slightly worse than getting data off raw disk,
>>>>> then its good enough.   Safely is supremely important, then its
>>>>> availability, then speed.
>>>>>>> Now, I think hbase is he most beautiful thing that happen to
>>>>>>> distributed DB world :).   The idea is to store image files
>>>>>>> 400Kb on average into HBASE).
>>>>>> I'd guess some images are much bigger than this.  Do you ever limit
>>>>>> the size of images folks can upload to your service?
>>>>>> The setup will include the following
>>>>>>> configuration:
>>>>>>> 50 servers total (2 datacenters), with 8 GB RAM, dual core cpu,
6 x
>>>>>>> 2TB disks each.
>>>>>>> 3 to 5 Zookeepers
>>>>>>> 2 Masters (in a datacenter each)
>>>>>>> 10 to 20 Stargate REST instances (one per server, hash loadbalanced)
>>>>>> Whats your frontend?  Why REST?  It might be more efficient if
>>>>>> could run with thrift given REST base64s its payload IIRC (check
>>>>>> src yourself).
>>>>> For insertion we use Haproxy, and balance curl PUTs across multiple REST
>>>>> For reading, its a nginx proxy that does Content-type modification
>>>>> from image/jpeg to octet-stream, and vice versa,
>>>>> it then hits Haproxy again, which hits balanced REST.
>>>>> Why REST, it was the simplest thing to run, given that its supports
>>>>> HTTP, potentially we could rewrite something for thrift, as long as we
>>>>> can use http still to send and receive data (anyone wrote anything
>>>>> like that say in python, C or java?)
>>>>>>> 40 to 50 RegionServers (will probably keep masters separate on
dedicated boxes).
>>>>>>> 2 Namenode servers (one backup, highly available, will do fsimage
>>>>>>> edits snapshots also)
>>>>>>> So far I got about 13 servers running, and doing about 20 insertions
>>>>>>> second (file size ranging from few KB to 2-3MB, ave. 400KB).
>>>>>>> Stargate API.  Our frontend servers receive files, and I just
>>>>>>> fork-insert them into stargate via http (curl).
>>>>>>> The inserts are humming along nicely, without any noticeable
load on
>>>>>>> regionservers, so far inserted about 2 TB worth of images.
>>>>>>> I have adjusted the region file size to be 512MB, and table block
>>>>>>> to about 400KB , trying to match average access block to limit
>>>>>>> trips.
>>>>>> As Todd suggests, I'd go up from 512MB... 1G at least.  You'll
>>>>>> probably want to up your flush size from 64MB to 128MB or maybe 192MB.
>>>>> Yep, i will adjust to 1G.  I thought flush was controlled by a
>>>>> function of memstore HEAP, something like 40%?  Or are you talking
>>>>> about HDFS block size?
>>>>>>  So far the read performance was more than adequate, and of
>>>>>>> course write performance is nowhere near capacity.
>>>>>>> So right now, all newly uploaded images go to HBASE.  But we
do plan
>>>>>>> to insert about 170 Million images (about 100 days worth), which
>>>>>>> only about 64 TB, or 10% of planned cluster size of 600TB.
>>>>>>> The end goal is to have a storage system that creates data safety,
>>>>>>> e.g. system may go down but data can not be lost.   Our Front-End
>>>>>>> servers will continue to serve images from their own file system
>>>>>>> are serving about 16 Gbits at peak), however should we need to
>>>>>>> any of those down for maintenance, we will redirect all traffic
>>>>>>> Hbase (should be no more than few hundred Mbps), while the front
>>>>>>> server is repaired (for example having its disk replaced), after
>>>>>>> repairs, we quickly repopulate it with missing files, while serving
>>>>>>> the missing remaining off Hbase.
>>>>>>> All in all should be very interesting project, and I am hoping
not to
>>>>>>> run into any snags, however, should that happens, I am pleased
to know
>>>>>>> that such a great and vibrant tech group exists that supports
and uses
>>>>>>> HBASE :).
>>>>>> We're definetly interested in how your project progresses.  If you
>>>>>> ever up in the city, you should drop by for a chat.
>>>>> Cool.  I'd like that.
>>>>>> St.Ack
>>>>>> P.S. I'm also w/ Todd that you should move to 0.89 and blooms.
>>>>>> P.P.S I updated the wiki on stargate REST:
>>>>>> http://wiki.apache.org/hadoop/Hbase/Stargate
>>>>> Cool, I assume if we move to that it won't kill existing meta tables,
>>>>> and data?  e.g. cross compatible?
>>>>> Is 0.89 ready for production environment?
>>>>> -Jack

View raw message