tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Braun <brianbr...@gmail.com>
Subject Best way to log requests from a servlet and to a database?
Date Sat, 26 Jan 2013 02:00:34 GMT
Hi,
First of all: I apologize because this subject is not specifically related
to Tomcat itself, but to be honest with you I dont know any other list with
people as nice and experienced as you guys are. And I bet many of you will
find this question very easy to answer.
Here it goes:

I am providing a RESTful service that is being served by a servlet (running
inside Tomcat 7.0.X and Ubuntu Linux). I'm already getting about 20
thousand queries per hour and it will grow much higher. The servlet
receives the requests, prepares the response, inserts a records in a MySQL
database table and delivers the response. The log in the database is
absolutely mandatory. Untily recently, all this happened in a syncronous
way. I mean, before the Tomcat thread delivered the response, it had to
create the records in the database table. The problem is that this log used
to take more than 90% of the total time, and even worse: when the database
got slower then the service took about 10-15 seconds instead of just 20
miliseconds.
I recetly made an improvement: Each Tomcat thread creates an extra thread
doing a "(new Thread(new certain Object)).start();" that takes care of the
SQL insertion in an asyncronous way, so the response gets to the clients
faster. But these threads take too much RAM when MySQL runs slower and
threads multiply, and with a few thousands of them the JVM Tomcat runs of
the memory.

What I need is to be able to accept as much HTTP requests as possible, to
log every one of them as fast as possible (not syncronously), and to make
everything fast and with a very low usage of RAM when MySQL gets slow and
inserts need to queue. I think I need some kind of queue to buffer the
entries when the speed of http request is higher than the speed of
insertions in the database log.

I'm thinking about these ideas:

1- Creating some kind of FIFO queue myself, maybe using some of those
Apache commons collections, and the some kind of thread that polls the
collection and creates the database records. But what collection should I
use? And how should I program the thread that polls it, so it won't
monopolize the CPU? I think that a "Do while (true)...." would eat the CPU
cycles. And that about making it thread safe? How to do it?
2- log4J? I have never used it directly, but it seems that this framework
is algo designed to creat "appenders" that talk to the database. Would that
be the way to do it?
3- Using some kind of any other framework that specializes in this?


What would you suggest?

Thanks in advance!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message