hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Tarnas <...@email.com>
Subject Re: EC2 + Thrift inserts
Date Fri, 30 Apr 2010 15:46:14 GMT
Thank you, I'll bump the handler higher and run a jstack on the most loaded one. Now I just
need more hours in the day to do it!

-chris

On Apr 29, 2010, at 9:14 PM, Ryan Rawson wrote:

> One thing to check is at the peak of your load, run jstack on one of
> the regionservers, and look at the handler threads - if all of them
> are doing something you might be running into handler contention.
> 
> it is basically ultimately IO bound.
> 
> -ryan
> 
> On Thu, Apr 29, 2010 at 9:12 PM, Chris Tarnas <cft@email.com> wrote:
>> They are all at 100, but none of the regionservers are loaded - most are
>> less than 20% CPU. Is this all network latency?
>> 
>> -chris
>> 
>> On Apr 29, 2010, at 8:29 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>> 
>>> Every insert on an indexed would require at the very least an RPC to a
>>> different regionserver.  If the regionservers are busy, your request
>>> could wait in the queue for a moment.
>>> 
>>> One param to tune would be the handler thread count.  Set it to 100 at
>>> least.
>>> 
>>> On Thu, Apr 29, 2010 at 2:16 AM, Chris Tarnas <cft@email.com> wrote:
>>>> 
>>>> I just finished some testing with JDK 1.6 u17 - so far no performance
>>>> improvements with just changing that. Disabling LZO compression did gain
a
>>>> little bit (up to about 30/sec from 25/sec per thread). Turning of indexes
>>>> helped the most - that brought me up to 115/sec @ 2875 total rows a second.
>>>> A single perl/thrift process can load at over 350 rows/sec so its not
>>>> scaling as well as I would have expected, even without the indexes.
>>>> 
>>>> Are the transactional indexes that costly? What is the bottleneck there?
>>>> CPU utilization and network packets went up when I disabled the indexes,
I
>>>> don't think those are the bottlenecks for the indexes. I was even able to
>>>> add another 15 insert process (total of 40) and only lost about 10% on a
per
>>>> process throughput. I probably could go even higher, none of the nodes are
>>>> above CPU 60% utilization and IO wait was at most 3.5%.
>>>> 
>>>> Each rowkey is unique, so there should not be any blocking on the row
>>>> locks. I'll do more indexed tests tomorrow.
>>>> 
>>>> thanks,
>>>> -chris
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Apr 29, 2010, at 12:18 AM, Todd Lipcon wrote:
>>>> 
>>>>> Definitely smells like JDK 1.6.0_18. Downgrade that back to 16 or 17
and
>>>>> you
>>>>> should be good to go. _18 is a botched release if I ever saw one.
>>>>> 
>>>>> -Todd
>>>>> 
>>>>> On Wed, Apr 28, 2010 at 10:54 PM, Chris Tarnas <cft@email.com>
wrote:
>>>>> 
>>>>>> Hi Stack,
>>>>>> 
>>>>>> Thanks for looking. I checked the ganglia charts, no server was at
more
>>>>>> than ~20% CPU utilization at any time during the load test and swap
was
>>>>>> never used. Network traffic was light - just running a count through
>>>>>> hbase
>>>>>> shell generates a much higher use. One the server hosting meta
>>>>>> specifically,
>>>>>> it was at about 15-20% CPU, and IO wait never went above 3%, was
>>>>>> usually
>>>>>> down at near 0.
>>>>>> 
>>>>>> The load also died with a thrift timeout on every single node (each
>>>>>> node
>>>>>> connecting to localhost for its thrift server), it looks like a
>>>>>> datanode
>>>>>> just died and caused every thrift connection to timeout - I'll have
to
>>>>>> up
>>>>>> that limit to handle a node death.
>>>>>> 
>>>>>> Checking logs this appears in the logs of the region server hosting
>>>>>> meta,
>>>>>> looks like the dead datanode causing this error:
>>>>>> 
>>>>>> 2010-04-29 01:01:38,948 WARN org.apache.hadoop.hdfs.DFSClient:
>>>>>> DFSOutputStream ResponseProcessor exception  for block
>>>>>> blk_508630839844593817_11180java.io.IOException: Bad response 1 for
>>>>>> block
>>>>>> blk_508630839844593817_11180 from datanode 10.195.150.255:50010
>>>>>>      at
>>>>>> 
>>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2423)
>>>>>> 
>>>>>> The regionserver log on teh dead node, 10.195.150.255 has some more
>>>>>> errors
>>>>>> in it:
>>>>>> 
>>>>>> http://pastebin.com/EFH9jz0w
>>>>>> 
>>>>>> I found this in the .out file on the datanode:
>>>>>> 
>>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode
>>>>>> linux-amd64 )
>>>>>> # Problematic frame:
>>>>>> # V  [libjvm.so+0x62263c]
>>>>>> #
>>>>>> # An error report file with more information is saved as:
>>>>>> # /usr/local/hadoop-0.20.1/hs_err_pid1364.log
>>>>>> #
>>>>>> # If you would like to submit a bug report, please visit:
>>>>>> #   http://java.sun.com/webapps/bugreport/crash.jsp
>>>>>> #
>>>>>> 
>>>>>> 
>>>>>> There is not a single error in the datanode's log though. Also of
note
>>>>>> -
>>>>>> this happened well into the test, so the node dying cause the load
to
>>>>>> abort
>>>>>> but not the prior poor performance. Looking through the mailing list
it
>>>>>> looks like java 1.6.0_18 has a bad rep so I'll update the AMI (although
>>>>>> I'm
>>>>>> using the same JVM on other servers in the office w/o issue and decent
>>>>>> single node performance and never dying...).
>>>>>> 
>>>>>> Thanks for any help!
>>>>>> -chris
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Apr 28, 2010, at 10:10 PM, Stack wrote:
>>>>>> 
>>>>>>> What is load on the server hosting meta like?  Higher than others?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Apr 28, 2010, at 8:42 PM, Chris Tarnas <cft@email.com>
wrote:
>>>>>>> 
>>>>>>>> Hi JG,
>>>>>>>> 
>>>>>>>> Speed is now down to 18 rows/sec/table per process.
>>>>>>>> 
>>>>>>>> Here is a regionserver log that is serving two of the regions:
>>>>>>>> 
>>>>>>>> http://pastebin.com/Hx5se0hz
>>>>>>>> 
>>>>>>>> Here is the GC Log from the same server:
>>>>>>>> 
>>>>>>>> http://pastebin.com/ChrRvxCx
>>>>>>>> 
>>>>>>>> Here is the master log:
>>>>>>>> 
>>>>>>>> http://pastebin.com/L1Kn66qU
>>>>>>>> 
>>>>>>>> The thrift server logs have nothing in them in the same time
period.
>>>>>>>> 
>>>>>>>> Thanks in advance!
>>>>>>>> 
>>>>>>>> -chris
>>>>>>>> 
>>>>>>>> On Apr 28, 2010, at 7:32 PM, Jonathan Gray wrote:
>>>>>>>> 
>>>>>>>>> Hey Chris,
>>>>>>>>> 
>>>>>>>>> That's a really significant slowdown.  I can't think
of anything
>>>>>> 
>>>>>> obvious that would cause that in your setup.
>>>>>>>>> 
>>>>>>>>> Any chance of some regionserver and master logs from
the time it was
>>>>>> 
>>>>>> going slow?  Is there any activity in the logs of the regionservers
>>>>>> hosting
>>>>>> the regions of the table being written to?
>>>>>>>>> 
>>>>>>>>> JG
>>>>>>>>> 
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Christopher Tarnas [mailto:cft@tarnas.org]
On Behalf Of Chris
>>>>>>>>>> Tarnas
>>>>>>>>>> Sent: Wednesday, April 28, 2010 6:27 PM
>>>>>>>>>> To: hbase-user@hadoop.apache.org
>>>>>>>>>> Subject: EC2 + Thrift inserts
>>>>>>>>>> 
>>>>>>>>>> Hello all,
>>>>>>>>>> 
>>>>>>>>>> First, thanks to all the HBase developers for producing
this, it's
>>>>>>>>>> a
>>>>>>>>>> great project and I'm glad to be able to use it.
>>>>>>>>>> 
>>>>>>>>>> I'm looking for some help and hints here with insert
performance
>>>>>>>>>> help.
>>>>>>>>>> I'm doing some benchmarking, testing how I can scale
up using
>>>>>>>>>> HBase,
>>>>>>>>>> not really looking at raw speed. The testing is happening
on EC2,
>>>>>> 
>>>>>> using
>>>>>>>>>> 
>>>>>>>>>> Andrew's scripts (thanks - those were very helpful)
to set them up
>>>>>>>>>> and
>>>>>>>>>> with a slightly customized version of the default
AMIs (added my
>>>>>>>>>> application modules). I'm using HBase 20.3 and Hadoop
20.1. I've
>>>>>> 
>>>>>> looked
>>>>>>>>>> 
>>>>>>>>>> at the tips in the Wiki and it looks like Andrew's
scripts are
>>>>>>>>>> already
>>>>>>>>>> setup that way.
>>>>>>>>>> 
>>>>>>>>>> I'm inserting into HBase from a hadoop streaming
job that runs perl
>>>>>> 
>>>>>> and
>>>>>>>>>> 
>>>>>>>>>> uses the thrift gateway. I'm also using the Transactional
tables so
>>>>>>>>>> that alone could be the case, but from what I can
tell I don't
>>>>>>>>>> think
>>>>>>>>>> so. LZO compression is also enabled for the column
families (much
>>>>>>>>>> of
>>>>>>>>>> the data is highly compressible). My cluster has
7 nodes, 5
>>>>>>>>>> regionservers, 1 master and 1 zookeeper. The regionservers
and
>>>>>>>>>> master
>>>>>>>>>> are c1.xlarges. Each regionserver has the tasktrackers
that runs
>>>>>>>>>> the
>>>>>>>>>> hadoop streaming jobs, and regionserver also runs
its own thrift
>>>>>>>>>> server. Each mapper that does the load talks to the
localhost's
>>>>>>>>>> thrift
>>>>>>>>>> server.
>>>>>>>>>> 
>>>>>>>>>> The Row keys a fixed string + an incremental number
then the order
>>>>>>>>>> of
>>>>>>>>>> the bytes are reversed, so runA123 becomes 321Anur.
I though of
>>>>>>>>>> using
>>>>>>>>>> murmur hash but was worried about collisions.
>>>>>>>>>> 
>>>>>>>>>> As I add more insert jobs, each jobs throughput goes
down. Way
>>>>>>>>>> down. I
>>>>>>>>>> went from about 200 row/sec/table per job with one
job to about 24
>>>>>>>>>> rows/sec/table per job with 25 running jobs. The
servers are mostly
>>>>>>>>>> idle. I'm loading into two tables, one has several
indexes and I'm
>>>>>>>>>> loading into three column families, the other has
no indexes and
>>>>>>>>>> one
>>>>>>>>>> column family. Both tables only currently have two
region each.
>>>>>>>>>> 
>>>>>>>>>> The regionserver that serves the indexed table's
regions is using
>>>>>>>>>> the
>>>>>>>>>> most CPU but is 87% idle. The other servers are all
at ~90% idle.
>>>>>> 
>>>>>> There
>>>>>>>>>> 
>>>>>>>>>> is no IO wait. the perl processes are barely ticking
over. Java on
>>>>>>>>>> the
>>>>>>>>>> most "loaded" server is using about 50-60% of one
CPU.
>>>>>>>>>> 
>>>>>>>>>> Normally when I do load in a pseudo-distrbuted hbase
(my
>>>>>>>>>> development
>>>>>>>>>> platform) perl's speed is the limiting factor and
uses about 85% of
>>>>>>>>>> a
>>>>>>>>>> CPU. In this cluster they are using only 5-10% of
a CPU as they are
>>>>>> 
>>>>>> all
>>>>>>>>>> 
>>>>>>>>>> waiting on thrift (hbase). When I run only 1 process
on the
>>>>>>>>>> cluster,
>>>>>>>>>> perl uses much more of a CPU, maybe 70%.
>>>>>>>>>> 
>>>>>>>>>> Any tips or help in getting the speed/scalability
up would be
>>>>>>>>>> great.
>>>>>>>>>> Please let me know if you need any other info.
>>>>>>>>>> 
>>>>>>>>>> As I send this - it looks like the main table has
split again and
>>>>>>>>>> is
>>>>>>>>>> being served by three regionservers.. My performance
is going up a
>>>>>>>>>> bit
>>>>>>>>>> (now 35 rows/sec/table per processes), but still
seems like I'm not
>>>>>>>>>> using the full potential of even the limited EC2
system, no IO wait
>>>>>> 
>>>>>> and
>>>>>>>>>> 
>>>>>>>>>> lots of idle CPU.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> many thanks
>>>>>>>>>> -chris
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>> 
>>>> 
>> 


Mime
View raw message