commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Béla Hullár <hull...@gmail.com>
Subject [vfs] Cache size issue
Date Wed, 13 Nov 2013 16:01:14 GMT
Hi All,

We've been developing a system where we move around a lot of files. We
started to use vfs, because sometimes these files are on a remote
filesystem and vfs is really a great library which enables us to handle
these cases uniform.

As the system was started to be used very heavily recently we experienced
memory issues, OutOfMemoryErrors. After investigating the problem it seems
the cause of the problem was the FilesCaches of the manager which was
growing without limits. We have few questions regarding this:

1, is this infinite growing of a SoftRefFilesCache a bug or a feature?
javadoc says if the jvm needs memory then the cache will free. But this one
by one removal in the maintenance thread of the cache doesn't seem to be
very efficient.

2, we started to use LRUFilesCache, because with that we can limit the
number of the cached entries. Indeed it seems it has much lower memory
usage (according the set limit), but checking a heap dump showed that our
LRUCache contained around ~21k entries however its size was set to 500. Do
you have some idea how it's possible?

I created a small test program which can reproduce the issue:
https://svn.code.sf.net/p/screeningbee/code/trunk/vfs-cache-test/src/main/java/ch/systemsx/bee/vfscachetest/CacheSizeChecker.java

if you set the testroot for a directory contains a lot of files and limit
the heapsize (eg. -Xmx128m) then it will fail with the default SoftRefCache
and with WeekRefCache and it works with LRU an NullFilesCache
Of course in the real application we have lot more memory, but it was also
eaten up after a while.

we use java 7 and vfs 2.0

Any help would be appreciated!
Bela

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message