db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bergquist, Brett" <BBergqu...@canoga.com>
Subject RE: Really need some help here please. I think I am having a problem with some sort of caching in Derby
Date Wed, 28 Mar 2012 14:51:21 GMT
Thanks Bryan.  I have tried that tool before and with the large heaps it has a problem.  

This is a network server setup and might be one of the more heavily used Derby applications:

- database size of about 50G of network device provisioning information and growing
- daily about 6.5M network performance records being added and the database grows to about
200Gb before being purged.   This has grown from about 1M since last June.
- a poor man's partitioning by putting network performance records into tables for each week
of the year (53 tables)
	- a View created to present these tables as one logical table that uses is based on a RestrictedTable
VTI that creates an optimizes a UNION query over the property tables dynamically based on
the query constraints
- continuous operation 7/24 including online backups done every night
- Web service interface that acts like tens to about 100 uses simultaneously accessing the
system
- Normally zero system administration is available

So far I have been able to tweak and tune and even fix some Derby bugs to keep this running
;)  We have definitely been stressing Derby!

Unfortunately out testing in our lab did not turn up these issues and it is only in production
at a customer site with zero access for me to monitor, that these problems have shown up.
 

But I think I am on the right track with the earlier email to Mike about the statement cache.
 And you are 100% right about getting a simpler setup if possible to recreate the problem.
 Once it is possible to recreate 100% of the time, I have no doubt a solution will be found.


Thanks for the input.

Brett

-----Original Message-----
From: Bryan Pendleton [mailto:bpendleton.derby@gmail.com] 
Sent: Wednesday, March 28, 2012 9:49 AM
To: derby-dev@db.apache.org
Subject: Re: Really need some help here please. I think I am having a problem with some sort
of caching in Derby

> So what I found is that if I hit the "Perform GC" button on JvisualVM 
> to perform an explicit garbage collection, the same query that was 
> taking 20 seconds or more instantaneously comes down to taking about 6 milliseconds.
Its processing time then starts to creep up back towards 20 seconds or more.

Is this an embedded configuration? Or are you looking at the Network Server?

If it's embedded, have you considered the possibility that it's something other than Derby
that's consuming the memory?

See if you can use a tool like this:

http://www.eclipse.org/mat/

to get some overall histogram of memory consumption before and after that explicit GC.

It might also make it easier to run your experiment under a smaller JVM, with a smaller workload,
because you'll have less overall data to record and the analysis tools will be more responsive.

But the patterns may be less clear.

thanks,

bryan



Mime
View raw message