esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Kohler <markus.koh...@gmail.com>
Subject ESME locking up during load test
Date Thu, 17 Dec 2009 09:40:56 GMT
Hi all,
I described Plan A and B last time for getting a first load test running,
but since I'm always told to be "agile" and "lean" I decided to go for plan
C ;-)

I've got the first load tests running using http://celerity.rubyforge.org/.
Celerity is an implemenation of watir (http://watir.com/) on top of Jruby
instead of "standard" ruby, which uses htmlunit (a Java library to simulate
a browser, without the rendering part) to execute the tests. Celerity
supports multithreaded tests and can therefore be used for load tests.

Since I have a some experience with watir, I thought  I could give it a try.
After I got around some issues caused by my limited understanding of how
multithreading works in ruby, it seems to work fine.

At least I got enough load  (50 users sending 10 messages) onto my local
ESME instance to lock it up. I first thought there's a dead lock, but It
would "only" respond very slowly.

This behavior seems to be caused by exceeding the available data base
connections. I'm still using Derby and I didn't check the configuration.

It seems that bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
retries to get a connection after 10 seconds.
It does a recursive call to do so and since no tail-recursion optimization
is kicking in I think this is not a good idea and should be changed.
The stacktrace is very long:

"32640132@qtp-18149218-105" prio=6 tid=0x4c4f5000 nid=0x1b34 in
Object.wait() [0
x4e19b000..0x4e19fc80]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at bootstrap.liftweb.DBVendor$.newConnection(Boot.scala:279)
        - locked <0x09682bd8> (a bootstrap.liftweb.DBVendor$)
        at
net.liftweb.mapper.DB$$anonfun$1$$anonfun$apply$1.apply(DB.scala:90)
        at
net.liftweb.mapper.DB$$anonfun$1$$anonfun$apply$1.apply(DB.scala:90)
        at net.liftweb.common.EmptyBox.or(Box.scala:374)
        at net.liftweb.mapper.DB$$anonfun$1.apply(DB.scala:90)
        at net.liftweb.mapper.DB$$anonfun$1.apply(DB.scala:90)
        at net.liftweb.common.Full.flatMap(Box.scala:332)
        at net.liftweb.mapper.DB$.newConnection(DB.scala:90)
        at net.liftweb.mapper.DB$.getConnection(DB.scala:186)
        at net.liftweb.mapper.DB$.use(DB.scala:446)
        at
net.liftweb.mapper.KeyedMetaMapper$class.findDbByKey(MetaMapper.scala
:1566)
        at org.apache.esme.model.User$.findDbByKey(User.scala:49)
        at
net.liftweb.mapper.KeyedMetaMapper$class.findDbByKey(MetaMapper.scala
:1562)
        at org.apache.esme.model.User$.findDbByKey(User.scala:49)
        at
net.liftweb.mapper.KeyedMetaMapper$$anonfun$find$2.apply(MetaMapper.s
cala:1547)
        at
net.liftweb.mapper.KeyedMetaMapper$$anonfun$find$2.apply(MetaMapper.s
cala:1547)
        at net.liftweb.common.Full.flatMap(Box.scala:332)
        at
net.liftweb.mapper.KeyedMetaMapper$class.find(MetaMapper.scala:1547)
        at org.apache.esme.model.User$.find(User.scala:49)
        at
org.apache.esme.model.User$curUser$$anonfun$$init$$2$$anonfun$apply$1
.apply(User.scala:172)
        at
org.apache.esme.model.User$curUser$$anonfun$$init$$2$$anonfun$apply$1
.apply(User.scala:172)
        at net.liftweb.common.Full.flatMap(Box.scala:332)



It seems we are hitting the DB to get the currentUser.
I have to redo the test and will then post the whole thread dump with all
the stack traces.
I can also open a JIRA ticket if you want that.


Regards,
Markus



"The best way to predict the future is to invent it" -- Alan Kay

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message