Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of nitingupta183@gmail.com
 designates 209.85.198.233 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=from:to:references:in-reply-to:subject:date:message-id:mime-version
         :content-type:content-transfer-encoding:x-mailer:thread-index
         :content-language;
        b=AyYRnB4+cu+pxi9/NbNgRjVFh1udriC2cXACW7RqHjz6i/YqIqZ26w0vY59FS8Uh6n
         Bqe5UkL+B02LhWbAKyD62izaQzkicEVNE+BzKfzFBebvbUYxoeYiCkFZVGeIlRiu4l13
         K9uWa2Preab2cObQ0fkE23PLQcGoq/MErcPW4=
From: "Nitin Gupta" <nitingupta183@gmail.com>
To: <hbase-user@hadoop.apache.org>
References: <cd61548c0906070644gecd6bb1i303851a761763ad8@mail.gmail.com>
 <56107.173.55.7.221.1244390379.squirrel@webmail.streamy.com>
In-Reply-To: <56107.173.55.7.221.1244390379.squirrel@webmail.streamy.com>
Subject: RE: Help needed - Adding HBase to architecture
Date: Mon, 8 Jun 2009 11:13:56 +0530
Message-ID: <003c01c9e7fc$2087df00$61979d00$@com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: AcnniQgl0X4tg8ylS26RKwGlBDy6lQAb0CAg
Content-Language: en-us

Jonathan,

Thanks for detailed explanation. Much helpful.

As far as file size is concerned, we may be even required to save Videos in
future. So we shall def go above the HBase size limit at some point in time.
Any other solution or key-value database that you can recommend for our
case?

I am not much knowledgeable about the HDFS either. I think if we go with
pure HDFS, then all the required DB operations would have to be custom
developed on top of HDFS. For our needs, do you think that HDFS already has
enough support that we will not need any major custom development. We are
just saving the files/attachements and retrieving them with some basic
search.

Regards,
Nitin

-----Original Message-----
From: Jonathan Gray [mailto:jlist@streamy.com] 
Sent: Sunday, June 07, 2009 9:30 PM
To: hbase-user@hadoop.apache.org
Subject: Re: Help needed - Adding HBase to architecture

Nitin,

HBase stores arbitrary binary values (row keys, column qualifiers, and
column values), so it is certainly capable of storing and serving files
and images.

My only real question before I would give you a +1 on your idea is what
you expect the range of file sizes to be.  While HBase allows you to store
values up to length Integer.MAX_VALUE, that is not recommended and in past
versions has lead to memory issues (OOME and such).

Images, text, word/excel docs, etc... should be no problem.  But I don't
recommend storing things in the upper 10s or 100s of MB, though it's
probably possible with a little work adjusting some configuration
parameters.  In general, if you are approaching HDFS block size, then you
really just want HDFS and not HBase :)

We are not currently running this in production, but we have had an
experimental version of our media server that runs on top of HBase rather
than the file system.  It has a series of Python scripts (connected to
HBase through our custom interface, you could use Java directly or
Thrift/REST/etc) that are responsible for generating various thumbnail
sizes.  The originals are stored in HBase, and then a special query is run
to grab the thumbnail of a certain size.  If it exists in HBase already,
it is just fetched and returned.  Otherwise, it is generated (via PIL,
Python Imaging Library, and some other custom tools), stored in HBase, and
then returned to the client.

As far as HBase on Windows goes... It's currently not possible but there
has been some effort from Powerset/Microsoft to make it happen.  I will
yield to those more familiar with it.

Personally, I run Windows on my primary work desktop and spend a good
chunk of my time on HBase development.  When I've wanted to spin up
pseudo-distributed local clusters, I usually use a cheap Linux node or
local Virtual Machine.  In both cases, I use a Windows X Server and
redirect output to my local Windows machine so I can run Eclipse and unit
tests from my Windows GUI.  Others have used Cygwin with some success, I
believe.

Hope that sheds some light for you.

You are almost certainly right about not wanting to store this in an
RDBMS.  And a hybrid approach seems to make sense, especially as a first
step.

Jonathan Gray


On Sun, June 7, 2009 6:44 am, Nitin Gupta wrote:
> Hi All,
> I am working on an application which is kind of a social network on mobile
>  WAP. Recently, we have incorporated the files or attachments support in
> our application. Right now, since we are not in production yet, we are
> keeping all the files in the RDBMS which our application is using. But I
> am more than convinvced that this is not going to work once we are in
> production mode.
>
> I got to know about HBase and I am making myself convice about its usage
> for the file storage, search and retrieval operations. I would like my
> opinion to be endorsed by expert HBase users/developers. Just for the
> clarification, here is what I am planning to do:
>
> Make use of a RDBMS for relational data in the application.
> All the files/blob data to be saved in the HBase.
> When required, my application can query app data from the RDBMS and the
> files can be retrieved from the HBase data store I will keep the meta data
> of the files in my rdbms so that files can be associated with my apps
> entities
>
> Please help me decide if this is the right approach. My app is supposed
> to provide support for images as well. So if anyone can advice if HBase is
> the right solution for me, in conjuction with an imaging tool.
>
> Since my team is predominantly Windows based, I would like to know is it
> possible to run HBase on a windows machine in stand alone and in clustered
>  mode.
>
> Thanks for all your help.
>
>
> nitin
>