hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Pinkham <jer...@lotame.com>
Subject RE: master performance
Date Tue, 02 Jun 2009 17:55:44 GMT
Thanks everyone for the feedback.  Here are the answers to the questions posed, and an update.

- the HTable is constructed during configure
- blockcache is enabled on the info family, but not the historian family.
- major compaction didn't have an impact
- a typical mapper in the job takes several minutes, how many minutes depends on whether I
use the the region partitioner and how many I let run concurrently... it's been anywhere from
2 minutes with no partitioner and small concurrency (5 mappers) to 8 minutes with the region
partitioner and high concurrency (150 mappers).  This seems to directly correlate with how
long it takes to do a simple count of .META. while each job is running (2 seconds to 1 minute)

Update:
I was able to get past this issue affecting my data load by reorganizing some of my workflow
and data structures to force the ordering of keys without the region partitioner.  Those changes
appear to have side stepped the problem for me as I can now load from 100+ mappers without
seeing the degradation that I was seeing with 40 when using the partitioner (and getting some
sweet numbers in the requests column of the UI).  It's still an interesting scaling situation
with the region partitioner, but I'm good to go without it.




-----Original Message-----
From: Jonathan Gray [mailto:jlist@streamy.com]
Sent: Tue 6/2/2009 1:36 AM
To: hbase-user@hadoop.apache.org
Subject: Re: master performance
 
And in your MapReduce jobs, make sure you are not instantiating more than
one HTable per task (there is a cost associated with it and can contribute
to load on .META.)

On Mon, June 1, 2009 8:53 pm, stack wrote:
> And check that you have block caching enabled on your .META. table.  Do
> "describe '.META.'" in the shell.  Its on by default but maybe you
> migrated from an older version or something else got in the way of its
> working.
>
> St.Ack
>
>
> On Mon, Jun 1, 2009 at 8:36 PM, stack <stack@duboce.net> wrote:
>
>
>> What Ryan said and then can you try same test after a major compaction?
>>  Does it make a difference?  You can force it in shell by doing "hbase>
>>  major_compaction '.META.'" IIRC (Type 'tools' in shell to get help
>> syntax).   What size are your jobs?  Short-lived?  Seconds or minutes?
>> Each
>> job needs to build up cache or region locations.  To do this, its trip
>> to .META.  Longer-lived jobs will save on trips to .META.  Also, take a
>> thread dump when its slow ("kill -QUIT PID_OF_MASTER") and send it to
>> us.  Do it a few times.  We'll take a look see.
>>
>> Should be better in 0.20.0 but maybe a few things we can do meantime.
>>
>>
>> St.Ack
>>
>>
>> On Mon, Jun 1, 2009 at 5:31 PM, Jeremy Pinkham <jeremy@lotame.com>
>> wrote:
>>
>>
>>>
>>> sorry for the novel...
>>>
>>> I've been experiencing some problems with my hbase cluster and hoping
>>>  someone can point me in the right direction.  I have a 40 node
>>> cluster running 0.19.0.  Each node has 4 cores, 8GB (2GB dedicated to
>>> the regionserver), and 1TB data disk.  The master is on a dedicated
>>> machine separate from the namenode and the jobtracker.  There is a
>>> single table with 4 column families and 3700 regions evenly spread
>>> across the 40 nodes.  The TTL's match our loading pace well enough
>>> that we don't typically see too many splits anymore.
>>>
>>> In trying to troubleshoot some larger issues with bulk loads on this
>>> cluster I have created a test scenario to try and narrow the problem
>>> based on various symptoms.  This test is map/reduce job that is using
>>> the HRegionPartitioner (as an easy way to generate some traffic to the
>>> master for meta data).  I've been running this job with various size
>>> inputs to gauge the effect of different numbers of mappers and have
>>> found that as the number of concurrent mappers creeps up to what I
>>> think are still small numbers (<50 mappers), the performance of the
>>> master is dramatically impacted.  I'm judging the performance here
>>> simply by checking the response time of the UI on the master, since
>>> that has historically been a good indication of when the cluster is
>>> getting into trouble during our loads (which I'm sure could mean a lot
>>> of things), although i suppose it's possible to two are unrelated.
>>>
>>> The UI normally takes about 5-7 seconds to refresh master.jsp.
>>> Running a
>>> job with 5 mappers doesn't seem to impact it too much, but a job with
>>> 38
>>> mappers makes the UI completely unresponsive for anywhere from 30
>>> seconds to several minutes during the run.  During this time, there is
>>> nothing happening in the logs, scans/gets from within the shell
>>> continue to work fine, and ganglia/top show the box to be virtually
>>> idle.  All links off of master.jsp work fine, so I presume it's
>>> something about the master pulling info from the individual nodes, but
>>> those UI's are also perfectly responsive.
>>>
>>> This same cluster used to run on just 20 nodes without issue, so I'm
>>> curious if I've crossed some threshold of horizontal scalability or if
>>> there is just a tuning parameter that I'm missing that might take care
>>> of this, or if there is something known between 0.19.0 and 0.19.3 that
>>> might be a factor.
>>>
>>> Thanks
>>>
>>>
>>> jeremy
>>>
>>>
>>> The information transmitted in this email is intended only for the
>>> person(s) or entity to which it is addressed and may contain
>>> confidential and/or privileged material. Any review, retransmission,
>>> dissemination or other use of, or taking of any action in reliance
>>> upon, this information by persons or entities other than the intended
>>> recipient is prohibited. If you received this email in error, please
>>> contact the sender and permanently delete the email from any computer.
>>>
>>>
>>>
>>
>









The information transmitted in this email is intended only for the person(s) or entity to
which it is addressed and may contain confidential and/or privileged material. Any review,
retransmission, dissemination or other use of, or taking of any action in reliance upon, this
information by persons or entities other than the intended recipient is prohibited. If you
received this email in error, please contact the sender and permanently delete the email from
any computer.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message