db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Matrigali <mikem_...@sbcglobal.net>
Subject Re: [jira] Closed: (DERBY-704) Large page cache kills initial performance
Date Wed, 16 Nov 2005 17:52:06 GMT
thanks, that makes sense - I just wasn't thinking about the
startup costs and that growing the cache created invalid pages in the
same way as "shrinking" does in the case of drop table.
I tend to ignore the startup and wait for the
steady state and concentrate on performance there, great
you found this issue.

The clock algorithm is an area that may be ripe for improvements
(or probably better a complete new cache factory),
especially when dealing with very large caches. Also the cache
may be the first place to look to use new more concurrent
data structures provided by java.  I would expect the current
design to scale reasonably well on 1, 2 and maybe 4 processors -
but it may see problems after that.  I would expect the most gain
to be first the buffer manager, next the lock manager, and then
the various other caches (statement cache, open file cache).

Knut Anders Hatlen wrote:
> Mike Matrigali <mikem_app@sbcglobal.net> writes:
>>Also I was wondering if you had any idea where the invalid pages
>>were coming from?
> The invalid pages are coming from Clock.growCache() where they are
> created but not initialized (hence invalid). The synchronization
> policy for Clock allows (to some extent) interleaving, so when there
> are more than one client there is a possibility that clients might see
> invalid pages not belonging to them. These pages are however marked
> with keepForCreate and cannot be taken by any other client than the
> one that created it.
>>The original problem with invalid pages was
>>from drop tables, where applications didn't like it that we used
>>all the pages in the cache when they were growing a single table
>>to 1/2 the cache size and then dropping it.  So the invalid case
>>was sort of special and wasn't expected to happen very often.
>>The usual case should be the cache growing to full and then LRU'ing
>>old pages to make for way for new ones.
>>Maybe there are some temp tables involved in your testing?
> No temp tables are involved. The load is really simple, with
> transactions consisting of one single-row select (using primary key),
> three single-row updates (using primary key) and one insert.
>>I thought you were doing a TPCC type of benchmark, so I am not
>>sure where the invalid pages are coming from.
>>Your work on big caches is very interesting, historically not much
>>work has been done in this area.
>>Mike Matrigali wrote:
>>>just got a chance to look at the patch, it would have been nice if
>>>some of the great comments that are in the bug description had made
>>>it into the code changes.
> You are right, I should have put more comments into the code. I'll see
> what I can do. Thanks for looking at the patch.

View raw message