hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Washusen <...@reactive.org>
Subject Re: Optimizations for random read performance
Date Wed, 17 Feb 2010 21:26:21 GMT
With a 12gb heap you now have 2.4GB for the block cache verses your original
800mb.  Out of interest, what are your block cache stats now?  Are they
above the 72% you were seeing previously?  Now that you have the resources,
it might also be worth while increasing the block cache from 0.2 to
something larger (like 0.4).

You also have more memory available for the OS level file system cache so
that probably helps.  Before adding more RAM you had pretty much all
available memory allocated to processes and nothing in the file system cache
(see the buffers section under Mem in top)...  Would you be able to send us
through the top output for the machines?

Another random thaught;  if you are not running map reduce task then you
could shut down the Job Trackers and put that memory to better use...

Anyway, thanks for sharing!

Cheers,
Dan

On 18 February 2010 06:31, Stack <stack@duboce.net> wrote:

> On Wed, Feb 17, 2010 at 11:04 AM, James Baldassari <james@dataxu.com>
> wrote:
> > OK, I'll do my best to capture our changes here.  Ideally we would have
> > changed one variable at a time, but since these performance problems
> > were happening in our production environment, we finally had to just
> > throw the kitchen sink at it.
>
> Been there.
>
>
>  So I'm not sure which combination of the
> > following fixed the problems, but hopefully this will be useful
> > nonetheless:
> >
> > - Upgraded Hadoop from 0.20 to 0.20.1 (Cloudera version
> > hadoop-0.20-0.20.1+169.56-1).  This version apparently has some fixes
> > for HDFS stability issues under load
> >
>
> I took a look in here
> http://archive.cloudera.com/cdh/2/hadoop-0.20.1+169.56.CHANGES.txt and
> nothing obvious jumps out.  I'm asking the cloudera lads if they think
> there is anything that could have made a difference.
>
> > - Applied the HBASE-2180 patch
> >
>
> I'd think this would have made a big difference (though you reported
> in earlier mail it had no effect).
>
> > - Decreased the hfile block size from 64k to 4k in the configuration
> > file, but we didn't alter our existing tables, so I'm not sure if this
> > change had any effect.
>
> Did you restart after making this change?  If so, the cluster picked
> it up.  You can verify by taking a look at a random hfile.  Do
> something like:
>
> ./bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -m PATH_TO_FILE_IN_HDFS
>
> This'll dump out meta stats on the chosen file part of which will be
> block size used writing the file.
>
> >
> > - Modified the Hadoop configuration to specify only one data directory
> > for HDFS and one Hadoop temp directory since our servers only have one
> > disk each.
> >
>
> I'd think this'd make for no difference in perf.
>
>
> > - Threw lots of hardware at it.  We upgraded the region servers (which
> > are also our HDFS data nodes) from 8-core/8G boxes to 16-core/16G.  Our
> > final system configuration is as follows:
> > 4x 16-core/16GB RAM/250GB SATAII HDFS data nodes / Hbase region servers
> > 1x 4-core/8GB RAM/250GB (RAID 1) SATAII namenode / Hbase master
> > 1x 4-core/8GB RAM/250GB (RAID 1) SATAII secondary namenode
> >
>
> This probably did it (smile).
>
> > - Increased region server max heap from 5G to 12G
> >
>
> So, all is in cache?  Is that what did fixed the issue?
>
> Want to send us regionserver logs?
>
> Whats the request rate you are servicing?  Whats it say on the master home
> page?
>
> > I think that's everything.  If I had to guess, I would say that
> > upgrading Hadoop and moving to bigger hardware with more heap space for
> > HBase was what did it.  If anyone wants more details, like some specific
> > config setting, let me know and I'll try to get that for you.
> >
> > HBase is having no problems keeping up with all the gets/puts now.  The
> > load on the region servers is evenly distributed and is very low (< 1).
> >
>
> Whats the utilization on these boxes?  Any cpu or i/o load?
>
> > Thanks again to everyone who helped me work through these issues.  I
> > really appreciate it.
> >
> Thanks for sticking with it.  Sorry for the loss of sleep.
> St.Ack
>
>
> >
> > On Wed, 2010-02-17 at 02:18 -0600, Daniel Washusen wrote:
> >> Glad you sorted it out!  Please do tell...
> >>
> >> On 17/02/2010, at 4:59 PM, James Baldassari <james@dataxu.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > I think we managed to solve our performance and load issues.
> >> > Everything
> >> > has been stable for about an hour now, but I'm not going to raise the
> >> > victory flag until the morning because we've had short periods of
> >> > stability in the past.
> >> >
> >> > I've been working on this problem non-stop for almost a week now, so I
> >> > really need to get some sleep, but if everything looks good tomorrow
> >> > I'll write up a summary of all the changes we made and share it with
> >> > the
> >> > group.  Hopefully this exercise in tuning for a high-throughput
> >> > real-time environment will be useful to others.
> >> >
> >> > Thanks,
> >> > James
> >> >
> >> >
> >> > On Tue, 2010-02-16 at 23:18 -0600, Stack wrote:
> >> >> When you look at top on the loaded server is it the regionserver or
> >> >> the datanode that is using up the cpu?
> >> >>
> >> >> I look at your hdfs listing.  Some of the regions have 3 and 4 files
> >> >> but most look fine.   A good few are on the compaction verge so I'd
> >> >> imagine a lot of compaction going on but this is background though
it
> >> >> does suck cpu and i/o... it shouldn't be too bad.
> >> >>
> >> >> I took a look at the regionserver log.  The server is struggling
> >> >> during which time period?  There is one log run at the start and
> >> >> there
> >> >> it seems like nothing untoward.  Please enable DEBUG going forward.
> >> >> It'll shed more light on whats going on: See
> >> >> http://wiki.apache.org/hadoop/Hbase/FAQ#A5 for how.  Otherwise, the
> >> >> log doesn't have anything  running long enough for it to have been
> >> >> under serious load.
> >> >>
> >> >> This is a four node cluster now?  You don't seem to have too many
> >> >> regions per server yet you have a pretty high read/write rate going
> >> >> by
> >> >> earlier requests postings.   Maybe you need to add more servers.  Are
> >> >> you going to add in those 16G machines?
> >> >>
> >> >> When you look at the master ui, you can see that the request rate
> >> >> over
> >> >> time is about the same for all regionservers?  (refresh the master
ui
> >> >> every so often to take a new sampling).
> >> >>
> >> >> St.Ack
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Tue, Feb 16, 2010 at 3:59 PM, James Baldassari
> >> >> <james@dataxu.com> wrote:
> >> >>> Nope.  We don't do any map reduce.  We're only using Hadoop for
> >> >>> HBase at
> >> >>> the moment.
> >> >>>
> >> >>> That one node, hdfs02, still has a load of 16 with around 40% I/O
> >> >>> and
> >> >>> 120% CPU.  The other nodes are all around 66% CPU with 0-1% I/O
> >> >>> and load
> >> >>> of 1 to 3.
> >> >>>
> >> >>> I don't think all the requests are going to hdfs02 based on the
> >> >>> status
> >> >>> 'detailed' output.  It seems like that node is just having a much
> >> >>> harder
> >> >>> time getting the data or something.  Maybe we have some incorrect
> >> >>> HDFS
> >> >>> setting.  All the configs are identical, though.
> >> >>>
> >> >>> -James
> >> >>>
> >> >>>
> >> >>> On Tue, 2010-02-16 at 17:45 -0600, Dan Washusen wrote:
> >> >>>> You mentioned in a previous email that you have a Task Tracker
> >> >>>> process
> >> >>>> running on each of the nodes.  Is there any chance there is
a map
> >> >>>> reduce job
> >> >>>> running?
> >> >>>>
> >> >>>> On 17 February 2010 10:31, James Baldassari <james@dataxu.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> On Tue, 2010-02-16 at 16:45 -0600, Stack wrote:
> >> >>>>>> On Tue, Feb 16, 2010 at 2:25 PM, James Baldassari <
> james@dataxu.com
> >> >>>>>> >
> >> >>>>> wrote:
> >> >>>>>>> On Tue, 2010-02-16 at 14:05 -0600, Stack wrote:
> >> >>>>>>>> On Tue, Feb 16, 2010 at 10:50 AM, James Baldassari
<
> james@dataxu.com
> >> >>>>>>>> >
> >> >>>>> wrote:
> >> >>>>>>>
> >> >>>>>>> Whether the keys themselves are evenly distributed
is another
> >> >>>>>>> matter.
> >> >>>>>>> Our keys are user IDs, and they should be fairly
random.  If
> >> >>>>>>> we do a
> >> >>>>>>> status 'detailed' in the hbase shell we see the
following
> >> >>>>>>> distribution
> >> >>>>>>> for the value of "requests" (not entirely sure
what this value
> >> >>>>>>> means):
> >> >>>>>>> hdfs01: 7078
> >> >>>>>>> hdfs02: 5898
> >> >>>>>>> hdfs03: 5870
> >> >>>>>>> hdfs04: 3807
> >> >>>>>>>
> >> >>>>>> That looks like they are evenly distributed.  Requests
are how
> >> >>>>>> many
> >> >>>>>> hits a second.  See the UI on master port 60010.  The
numbers
> >> >>>>>> should
> >> >>>>>> match.
> >> >>>>>
> >> >>>>> So the total across all 4 region servers would be 22,653/
> >> >>>>> second?  Hmm,
> >> >>>>> that doesn't seem too bad.  I guess we just need a little
more
> >> >>>>> throughput...
> >> >>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>> There are no order of magnitude differences here,
and the
> >> >>>>>>> request count
> >> >>>>>>> doesn't seem to map to the load on the server.
 Right now
> >> >>>>>>> hdfs02 has a
> >> >>>>>>> load of 16 while the 3 others have loads between
1 and 2.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> This is interesting.  I went back over your dumps of
cache
> >> >>>>>> stats above
> >> >>>>>> and the 'loaded' servers didn't have any attribute
there that
> >> >>>>>> differentiated it from others.  For example, the number
of
> >> >>>>>> storefiles
> >> >>>>>> seemed about same.
> >> >>>>>>
> >> >>>>>> I wonder what is making for the high load?  Can you
figure it?
> >> >>>>>> Is it
> >> >>>>>> high CPU use (unlikely).  Is it then high i/o?  Can
you try and
> >> >>>>>> figure
> >> >>>>>> whats different about the layout under the loaded server
and
> >> >>>>>> that of
> >> >>>>>> an unloaded server?  Maybe do a ./bin/hadoop fs -lsr
/hbase and
> >> >>>>>> see if
> >> >>>>>> anything jumps out at you.
> >> >>>>>
> >> >>>>> It's I/O wait that is killing the highly loaded server.
 The CPU
> >> >>>>> usage
> >> >>>>> reported by top is just about the same across all servers
> >> >>>>> (around 100%
> >> >>>>> on an 8-core node), but one server at any given time has
a much
> >> >>>>> higher
> >> >>>>> load due to I/O.
> >> >>>>>
> >> >>>>>>
> >> >>>>>> If you want to post the above or a loaded servers log
to
> >> >>>>>> pastbin we'll
> >> >>>>>> take a looksee.
> >> >>>>>
> >> >>>>> I'm not really sure what to look for, but maybe someone
else
> >> >>>>> will notice
> >> >>>>> something, so here's the output of hadoop fs -lsr /hbase:
> >> >>>>> http://pastebin.com/m98096de
> >> >>>>>
> >> >>>>> And here is today's region server log from hdfs02, which
seems
> >> >>>>> to get
> >> >>>>> hit particularly hard: http://pastebin.com/m1d8a1e5f
> >> >>>>>
> >> >>>>> Please note that we restarted it several times today, so
some of
> >> >>>>> those
> >> >>>>> errors are probably just due to restarting the region server.
> >> >>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> Applying
> >> >>>>>>> HBASE-2180 did not make any measurable difference.
 There are
> >> >>>>>>> no errors
> >> >>>>>>> in the region server logs.  However, looking at
the Hadoop
> >> >>>>>>> datanode
> >> >>>>>>> logs, I'm seeing lots of these:
> >> >>>>>>>
> >> >>>>>>> 2010-02-16 17:07:54,064 ERROR
> >> >>>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
> >> >>>>> DatanodeRegistration(
> >> >>>>> 10.24.183.165:50010,
> >> >>>>> storageID=DS-1519453437-10.24.183.165-50010-1265907617548,
> >> >>>>> infoPort=50075,
> >> >>>>> ipcPort=50020):DataXceiver
> >> >>>>>>> java.io.EOFException
> >> >>>>>>>       at java.io.DataInputStream.readShort
> >> >>>>>>> (DataInputStream.java:298)
> >> >>>>>>>       at
> >> >>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run
> >> >>>>> (DataXceiver.java:79)
> >> >>>>>>>       at java.lang.Thread.run(Thread.java:619)
> >> >>>>>>
> >> >>>>>> You upped xceivers on your hdfs cluster?  If you look
at
> >> >>>>>> otherend of
> >> >>>>>> the above EOFE, can you see why it died?
> >> >>>>>
> >> >>>>> Max xceivers = 3072; datanode handler count = 20; region
server
> >> >>>>> handler
> >> >>>>> count = 100
> >> >>>>>
> >> >>>>> I can't find the other end of the EOFException.  I looked
in the
> >> >>>>> Hadoop
> >> >>>>> and HBase logs on the server that is the name node and
HBase
> >> >>>>> master, as
> >> >>>>> well as the on HBase client.
> >> >>>>>
> >> >>>>> Thanks for all the help!
> >> >>>>>
> >> >>>>> -James
> >> >>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>>
> >> >>>>>>> However, I do think it's strange that
> >> >>>>>>> the load is so unbalanced on the region servers.
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>> I agree.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>> We're also going to try throwing some more hardware
at the
> >> >>>>>>> problem.
> >> >>>>>>> We'll set up a new cluster with 16-core, 16G nodes
to see if
> >> >>>>>>> they are
> >> >>>>>>> better able to handle the large number of client
requests.  We
> >> >>>>>>> might
> >> >>>>>>> also decrease the block size to 32k or lower.
> >> >>>>>>>
> >> >>>>>> Ok.
> >> >>>>>>
> >> >>>>>>>> Should only be a matter if you intend distributing
the above.
> >> >>>>>>>
> >> >>>>>>> This is probably a topic for a separate thread,
but I've never
> >> >>>>>>> seen a
> >> >>>>>>> legal definition for the word "distribution." 
How does this
> >> >>>>>>> apply to
> >> >>>>>>> the SaaS model?
> >> >>>>>>>
> >> >>>>>> Fair enough.
> >> >>>>>>
> >> >>>>>> Something is up.  Especially if hbase-2180 made no
difference.
> >> >>>>>>
> >> >>>>>> St.Ack
> >> >>>>>
> >> >>>>>
> >> >>>
> >> >>>
> >> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message