Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of adrien.mogenet@gmail.com
 designates 209.85.220.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAHyy+G9EN8miV+DRg7xS5=mh2TEh_91iWKd=10kmxyXaEbdmng@mail.gmail.com>
References: <-135344591883378152@unknownmsgid>
	<CAKxWWm1sU0riT+4nYtyw72v2QvW+x+o8E9tger7irmUB-iNS-A@mail.gmail.com>
	<CAHyy+G879dreY_rSwo1Txy2Pg7cr3Z9S7g1uaEw8mPmSfK7n1A@mail.gmail.com>
	<CAKxWWm3J2Ge22acyabtwis7QMnOcs5-GPvLmw6BqJ7hXd5jnLw@mail.gmail.com>
	<CAHyy+G9AbiMW9NOkt16dwOSRt==prLR3cybEN4rEYsZOTfGL2g@mail.gmail.com>
	<CAG2rwuPekfEDYEVvs8XwsX33yw9bgOc2EQ=WD09zumiRUjqubw@mail.gmail.com>
	<CAHyy+G-bOfZwFMf3drT44TK6CqNTSwxkqx+=gUJvstYsMTdDYw@mail.gmail.com>
	<CAG2rwuMce_d5-6wYxeDqwdYwbvWnB=2shwUimJUfKn1SQE_iJQ@mail.gmail.com>
	<CAHyy+G9EN8miV+DRg7xS5=mh2TEh_91iWKd=10kmxyXaEbdmng@mail.gmail.com>
Date: Mon, 28 Jan 2013 11:01:00 +0100
Message-ID: 
 <CADt=Q7naVtQDTn-cT8ro8WdOKSrwHNrcxhcQtKx7pLaxwze0Vg@mail.gmail.com>
Subject: Re: Storing images in Hbase
From: Adrien Mogenet <adrien.mogenet@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=047d7b2ede71b35d1704d4565af5

--047d7b2ede71b35d1704d4565af5
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Could HCatalog be an option ?
Le 26 janv. 2013 21:56, "Jack Levin" <magnito@gmail.com> a =C3=A9crit :
>
> AFAIK, namenode would not like tracking 20 billion small files :)
>
> -jack
>
> On Sat, Jan 26, 2013 at 6:00 PM, S Ahmed <sahmed1020@gmail.com> wrote:
> > That's pretty amazing.
> >
> > What I am confused is, why did you go with hbase and not just straight
into
> > hdfs?
> >
> >
> >
> >
> > On Fri, Jan 25, 2013 at 2:41 AM, Jack Levin <magnito@gmail.com> wrote:
> >
> >> Two people including myself, its fairly hands off. Took about 3 months
to
> >> tune it right, however we did have had multiple years of experience
with
> >> datanodes and hadoop in general, so that was a good boost.
> >>
> >> We have 4 hbase clusters today, image store being largest
> >> On Jan 24, 2013 2:14 PM, "S Ahmed" <sahmed1020@gmail.com> wrote:
> >>
> >> > Jack, out of curiosity, how many people manage the hbase related
servers?
> >> >
> >> > Does it require constant monitoring or its fairly hands-off now?
 (or a
> >> bit
> >> > of both, early days was getting things write/learning and now its
purring
> >> > along).
> >> >
> >> >
> >> > On Wed, Jan 23, 2013 at 11:53 PM, Jack Levin <magnito@gmail.com>
wrote:
> >> >
> >> > > Its best to keep some RAM for caching of the filesystem, besides w=
e
> >> > > also run datanode which takes heap as well.
> >> > > Now, please keep in mind that even if you specify heap of say 5GB,
if
> >> > > your server opens threads to communicate with other systems via RP=
C
> >> > > (which hbase does a lot), you will indeed use HEAP +
> >> > > Nthreads*thread*kb_size.  There is a good Sun Microsystems documen=
t
> >> > > about it. (I don't have the link handy).
> >> > >
> >> > > -Jack
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Jan 21, 2013 at 5:10 PM, Varun Sharma <varun@pinterest.com=
>
> >> > wrote:
> >> > > > Thanks for the useful information. I wonder why you use only 5G
heap
> >> > when
> >> > > > you have an 8G machine ? Is there a reason to not use all of it
(the
> >> > > > DataNode typically takes a 1G of RAM)
> >> > > >
> >> > > > On Sun, Jan 20, 2013 at 11:49 AM, Jack Levin <magnito@gmail.com>
> >> > wrote:
> >> > > >
> >> > > >> I forgot to mention that I also have this setup:
> >> > > >>
> >> > > >> <property>
> >> > > >>   <name>hbase.hregion.memstore.flush.size</name>
> >> > > >>   <value>33554432</value>
> >> > > >>   <description>Flush more often. Default: 67108864</description=
>
> >> > > >> </property>
> >> > > >>
> >> > > >> This parameter works on per region amount, so this means if any
of
> >> my
> >> > > >> 400 (currently) regions on a regionserver has 30MB+ in
memstore, the
> >> > > >> hbase will flush it to disk.
> >> > > >>
> >> > > >>
> >> > > >> Here are some metrics from a regionserver:
> >> > > >>
> >> > > >> requests=3D2, regions=3D370, stores=3D370, storefiles=3D1390,
> >> > > >> storefileIndexSize=3D304, memstoreSize=3D2233,
compactionQueueSize=3D0,
> >> > > >> flushQueueSize=3D0, usedHeap=3D3516, maxHeap=3D4987,
> >> > > >> blockCacheSize=3D790656256, blockCacheFree=3D255245888,
> >> > > >> blockCacheCount=3D2436, blockCacheHitCount=3D218015828,
> >> > > >> blockCacheMissCount=3D13514652, blockCacheEvictedCount=3D256151=
6,
> >> > > >> blockCacheHitRatio=3D94, blockCacheHitCachingRatio=3D98
> >> > > >>
> >> > > >> Note, that memstore is only 2G, this particular regionserver
HEAP is
> >> > set
> >> > > >> to 5G.
> >> > > >>
> >> > > >> And last but not least, its very important to have good GC
setup:
> >> > > >>
> >> > > >> export HBASE_OPTS=3D"$HBASE_OPTS -verbose:gc -Xms5000m
> >> > > >> -XX:CMSInitiatingOccupancyFraction=3D70 -XX:+PrintGCDetails
> >> > > >> -XX:+PrintGCDateStamps
> >> > > >> -XX:+HeapDumpOnOutOfMemoryError
> >> -Xloggc:$HBASE_HOME/logs/gc-hbase.log
> >> > \
> >> > > >> -XX:MaxTenuringThreshold=3D15 -XX:SurvivorRatio=3D8 \
> >> > > >> -XX:+UseParNewGC \
> >> > > >> -XX:NewSize=3D128m -XX:MaxNewSize=3D128m \
> >> > > >> -XX:-UseAdaptiveSizePolicy \
> >> > > >> -XX:+CMSParallelRemarkEnabled \
> >> > > >> -XX:-TraceClassUnloading
> >> > > >> "
> >> > > >>
> >> > > >> -Jack
> >> > > >>
> >> > > >> On Thu, Jan 17, 2013 at 3:29 PM, Varun Sharma <
varun@pinterest.com>
> >> > > wrote:
> >> > > >> > Hey Jack,
> >> > > >> >
> >> > > >> > Thanks for the useful information. By flush size being 15 %,
do
> >> you
> >> > > mean
> >> > > >> > the memstore flush size ? 15 % would mean close to 1G, have
you
> >> seen
> >> > > any
> >> > > >> > issues with flushes taking too long ?
> >> > > >> >
> >> > > >> > Thanks
> >> > > >> > Varun
> >> > > >> >
> >> > > >> > On Sun, Jan 13, 2013 at 8:17 AM, Jack Levin <magnito@gmail.co=
m
>
> >> > > wrote:
> >> > > >> >
> >> > > >> >> That's right, Memstore size , not flush size is increased.
> >> >  Filesize
> >> > > is
> >> > > >> >> 10G. Overall write cache is 60% of heap and read cache is
20%.
> >> >  Flush
> >> > > >> size
> >> > > >> >> is 15%.  64 maxlogs at 128MB. One namenode server, one
secondary
> >> > that
> >> > > >> can
> >> > > >> >> be promoted.  On the way to hbase images are written to a
queue,
> >> so
> >> > > >> that we
> >> > > >> >> can take Hbase down for maintenance and still do inserts
later.
> >> > > >>  ImageShack
> >> > > >> >> has =E2=80=98perma cache=E2=80=99 servers that allows writes=
 and serving of
data
> >> > even
> >> > > >> when
> >> > > >> >> hbase is down for hours, consider it 4th replica =F0=9F=98=
=89 outside
of
> >> > hadoop
> >> > > >> >>
> >> > > >> >> Jack
> >> > > >> >>
> >> > > >> >>  *From:* Mohit Anchlia <mohitanchlia@gmail.com>
> >> > > >> >> *Sent:* =E2=80=8EJanuary=E2=80=8E =E2=80=8E13=E2=80=8E, =E2=
=80=8E2013 =E2=80=8E7=E2=80=8E:=E2=80=8E48=E2=80=8E =E2=80=8EAM
> >> > > >> >> *To:* user@hbase.apache.org
> >> > > >> >> *Subject:* Re: Storing images in Hbase
> >> > > >> >>
> >> > > >> >> Thanks Jack for sharing this information. This definitely
makes
> >> > sense
> >> > > >> when
> >> > > >> >> using the type of caching layer. You mentioned about
increasing
> >> > write
> >> > > >> >> cache, I am assuming you had to increase the following
parameters
> >> > in
> >> > > >> >> addition to increase the memstore size:
> >> > > >> >>
> >> > > >> >> hbase.hregion.max.filesize
> >> > > >> >> hbase.hregion.memstore.flush.size
> >> > > >> >>
> >> > > >> >> On Fri, Jan 11, 2013 at 9:47 AM, Jack Levin <
magnito@gmail.com>
> >> > > wrote:
> >> > > >> >>
> >> > > >> >> > We buffer all accesses to HBASE with Varnish SSD based
caching
> >> > > layer.
> >> > > >> >> > So the impact for reads is negligible.  We have 70 node
> >> cluster,
> >> > 8
> >> > > GB
> >> > > >> >> > of RAM per node, relatively weak nodes (intel core 2 duo),
with
> >> > > >> >> > 10-12TB per server of disks.  Inserting 600,000 images per
day.
> >> >  We
> >> > > >> >> > have relatively little of compaction activity as we made
our
> >> > write
> >> > > >> >> > cache much larger than read cache - so we don't experience
> >> region
> >> > > file
> >> > > >> >> > fragmentation as much.
> >> > > >> >> >
> >> > > >> >> > -Jack
> >> > > >> >> >
> >> > > >> >> > On Fri, Jan 11, 2013 at 9:40 AM, Mohit Anchlia <
> >> > > >> mohitanchlia@gmail.com>
> >> > > >> >> > wrote:
> >> > > >> >> > > I think it really depends on volume of the traffic, data
> >> > > >> distribution
> >> > > >> >> per
> >> > > >> >> > > region, how and when files compaction occurs, number of
nodes
> >> > in
> >> > > the
> >> > > >> >> > > cluster. In my experience when it comes to blob data
where
> >> you
> >> > > are
> >> > > >> >> > serving
> >> > > >> >> > > 10s of thousand+ requests/sec writes and reads then it's
very
> >> > > >> difficult
> >> > > >> >> > to
> >> > > >> >> > > manage HBase without very hard operations and
maintenance in
> >> > > play.
> >> > > >> Jack
> >> > > >> >> > > earlier mentioned they have 1 billion images, It would b=
e
> >> > > >> interesting
> >> > > >> >> to
> >> > > >> >> > > know what they see in terms of compaction, no of
requests per
> >> > > sec.
> >> > > >> I'd
> >> > > >> >> be
> >> > > >> >> > > surprised that in high volume site it can be done
without any
> >> > > >> Caching
> >> > > >> >> > layer
> >> > > >> >> > > on the top to alleviate IO spikes that occurs because of
GC
> >> and
> >> > > >> >> > compactions.
> >> > > >> >> > >
> >> > > >> >> > > On Fri, Jan 11, 2013 at 7:27 AM, Mohammad Tariq <
> >> > > dontariq@gmail.com
> >> > > >> >
> >> > > >> >> > wrote:
> >> > > >> >> > >
> >> > > >> >> > >> IMHO, if the image files are not too huge, Hbase can
> >> > efficiently
> >> > > >> serve
> >> > > >> >> > the
> >> > > >> >> > >> purpose. You can store some additional info along with
the
> >> > file
> >> > > >> >> > depending
> >> > > >> >> > >> upon your search criteria to make the search faster.
Say if
> >> > you
> >> > > >> want
> >> > > >> >> to
> >> > > >> >> > >> fetch images by the type, you can store images in one
column
> >> > and
> >> > > >> its
> >> > > >> >> > >> extension in another column(jpg, tiff etc).
> >> > > >> >> > >>
> >> > > >> >> > >> BTW, what exactly is the problem which you are facing.
You
> >> > have
> >> > > >> >> written
> >> > > >> >> > >> "But I still cant do it"?
> >> > > >> >> > >>
> >> > > >> >> > >> Warm Regards,
> >> > > >> >> > >> Tariq
> >> > > >> >> > >> https://mtariq.jux.com/
> >> > > >> >> > >>
> >> > > >> >> > >>
> >> > > >> >> > >> On Fri, Jan 11, 2013 at 8:30 PM, Michael Segel <
> >> > > >> >> > michael_segel@hotmail.com
> >> > > >> >> > >> >wrote:
> >> > > >> >> > >>
> >> > > >> >> > >> > That's a viable option.
> >> > > >> >> > >> > HDFS reads are faster than HBase, but it would requir=
e
> >> first
> >> > > >> hitting
> >> > > >> >> > the
> >> > > >> >> > >> > index in HBase which points to the file and then
fetching
> >> > the
> >> > > >> file.
> >> > > >> >> > >> > It could be faster... we found storing binary data in
a
> >> > > sequence
> >> > > >> >> file
> >> > > >> >> > and
> >> > > >> >> > >> > indexed on HBase to be faster than HBase, however,
YMMV
> >> and
> >> > > HBase
> >> > > >> >> has
> >> > > >> >> > >> been
> >> > > >> >> > >> > improved since we did that project....
> >> > > >> >> > >> >
> >> > > >> >> > >> >
> >> > > >> >> > >> > On Jan 10, 2013, at 10:56 PM, shashwat shriparv <
> >> > > >> >> > >> dwivedishashwat@gmail.com>
> >> > > >> >> > >> > wrote:
> >> > > >> >> > >> >
> >> > > >> >> > >> > > Hi Kavish,
> >> > > >> >> > >> > >
> >> > > >> >> > >> > > i have a better idea for you copy your image files
to a
> >> > > single
> >> > > >> >> file
> >> > > >> >> > on
> >> > > >> >> > >> > > hdfs, and if new image comes append it to the
existing
> >> > > image,
> >> > > >> and
> >> > > >> >> > keep
> >> > > >> >> > >> > and
> >> > > >> >> > >> > > update the metadata and the offset to the HBase.
Because
> >> > if
> >> > > you
> >> > > >> >> put
> >> > > >> >> > >> > bigger
> >> > > >> >> > >> > > image in hbase it wil lead to some issue.
> >> > > >> >> > >> > >
> >> > > >> >> > >> > >
> >> > > >> >> > >> > >
> >> > > >> >> > >> > > =E2=88=9E
> >> > > >> >> > >> > > Shashwat Shriparv
> >> > > >> >> > >> > >
> >> > > >> >> > >> > >
> >> > > >> >> > >> > >
> >> > > >> >> > >> > > On Fri, Jan 11, 2013 at 9:21 AM, lars hofhansl <
> >> > > >> larsh@apache.org>
> >> > > >> >> > >> wrote:
> >> > > >> >> > >> > >
> >> > > >> >> > >> > >> Interesting. That's close to a PB if my math is
> >> correct.
> >> > > >> >> > >> > >> Is there a write up about this somewhere? Somethin=
g
> >> that
> >> > we
> >> > > >> could
> >> > > >> >> > link
> >> > > >> >> > >> > >> from the HBase homepage?
> >> > > >> >> > >> > >>
> >> > > >> >> > >> > >> -- Lars
> >> > > >> >> > >> > >>
> >> > > >> >> > >> > >>
> >> > > >> >> > >> > >> ----- Original Message -----
> >> > > >> >> > >> > >> From: Jack Levin <magnito@gmail.com>
> >> > > >> >> > >> > >> To: user@hbase.apache.org
> >> > > >> >> > >> > >> Cc: Andrew Purtell <apurtell@apache.org>
> >> > > >> >> > >> > >> Sent: Thursday, January 10, 2013 9:24 AM
> >> > > >> >> > >> > >> Subject: Re: Storing images in Hbase
> >> > > >> >> > >> > >>
> >> > > >> >> > >> > >> We stored about 1 billion images into hbase with
file
> >> > size
> >> > > up
> >> > > >> to
> >> > > >> >> > 10MB.
> >> > > >> >> > >> > >> Its been running for close to 2 years without
issues
> >> and
> >> > > >> serves
> >> > > >> >> > >> > >> delivery of images for Yfrog and ImageShack.  If
you
> >> have
> >> > > any
> >> > > >> >> > >> > >> questions about the setup, I would be glad to
answer
> >> > them.
> >> > > >> >> > >> > >>
> >> > > >> >> > >> > >> -Jack
> >> > > >> >> > >> > >>
> >> > > >> >> > >> > >> On Sun, Jan 6, 2013 at 1:09 PM, Mohit Anchlia <
> >> > > >> >> > mohitanchlia@gmail.com
> >> > > >> >> > >> >
> >> > > >> >> > >> > >> wrote:
> >> > > >> >> > >> > >>> I have done extensive testing and have found that
> >> blobs
> >> > > don't
> >> > > >> >> > belong
> >> > > >> >> > >> in
> >> > > >> >> > >> > >> the
> >> > > >> >> > >> > >>> databases but are rather best left out on the fil=
e
> >> > system.
> >> > > >> >> Andrew
> >> > > >> >> > >> > >> outlined
> >> > > >> >> > >> > >>> issues that you'll face and not to mention IO
issues
> >> > when
> >> > > >> >> > compaction
> >> > > >> >> > >> > >> occurs
> >> > > >> >> > >> > >>> over large files.
> >> > > >> >> > >> > >>>
> >> > > >> >> > >> > >>> On Sun, Jan 6, 2013 at 12:52 PM, Andrew Purtell <
> >> > > >> >> > apurtell@apache.org
> >> > > >> >> > >> >
> >> > > >> >> > >> > >> wrote:
> >> > > >> >> > >> > >>>
> >> > > >> >> > >> > >>>> I meant this to say "a few really large values"
> >> > > >> >> > >> > >>>>
> >> > > >> >> > >> > >>>> On Sun, Jan 6, 2013 at 12:49 PM, Andrew Purtell =
<
> >> > > >> >> > >> apurtell@apache.org>
> >> > > >> >> > >> > >>>> wrote:
> >> > > >> >> > >> > >>>>
> >> > > >> >> > >> > >>>>> Consider if the split threshold is 2 GB but
your one
> >> > row
> >> > > >> >> > contains
> >> > > >> >> > >> 10
> >> > > >> >> > >> > >> GB
> >> > > >> >> > >> > >>>> as
> >> > > >> >> > >> > >>>>> really large value.
> >> > > >> >> > >> > >>>>
> >> > > >> >> > >> > >>>>
> >> > > >> >> > >> > >>>>
> >> > > >> >> > >> > >>>>
> >> > > >> >> > >> > >>>> --
> >> > > >> >> > >> > >>>> Best regards,
> >> > > >> >> > >> > >>>>
> >> > > >> >> > >> > >>>>   - Andy
> >> > > >> >> > >> > >>>>
> >> > > >> >> > >> > >>>> Problems worthy of attack prove their worth by
> >> hitting
> >> > > >> back. -
> >> > > >> >> > Piet
> >> > > >> >> > >> > Hein
> >> > > >> >> > >> > >>>> (via Tom White)
> >> > > >> >> > >> > >>>>
> >> > > >> >> > >> > >>
> >> > > >> >> > >> > >>
> >> > > >> >> > >> >
> >> > > >> >> > >> >
> >> > > >> >> > >>
> >> > > >> >> >
> >> > > >> >>
> >> > > >>
> >> > >
> >> >
> >>

--047d7b2ede71b35d1704d4565af5--