db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lars Clausen ...@statsbiblioteket.dk>
Subject Re: Workarounds for too many open files?
Date Mon, 28 Nov 2005 10:10:25 GMT
On Fri, 2005-11-25 at 09:14, Lars Clausen wrote:
> Trying to import a 10GB text file (about 50x10^6 entries) into a single
> Derby table, I got the following error:
> 
> ij> connect 'jdbc:derby:cdxdb'; ij> elapsedtime on; ij> CALL
> SYSCS_UTIL.SYSCS_IMPORT_DATA ( null, 'CDX',
> 'URL,IP,MIMETYPE,LENGTH,ARCFILE,OFFSET', '1,2,4,5,6,7',
> '/home/lc/index-backping.cdx', '`', null, null, 1);
> ERROR 38000: The exception 'SQL Exception: Exception during creation of
> file
> /home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132842374093.tmp
> for container' was thrown while evaluating an expression.
> ERROR XSDF1: Exception during creation of
> file
> /home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132842374093.tmp
> for container
> ERROR XJ001: Java exception:
> '/home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132842374093.tmp (Too many open
files): java.io.FileNotFoundException'.

It turns out that this happens during index creation.  I was able to
import the text file and run selects on it, but when I try to create an
index:

ij> select count(*) from cdx;
1
-----------
50000000
 
1 row selected
ELAPSED TIME = 320818 milliseconds
ij> create index cdxurl on cdx(url);
ERROR XSDF1: Exception during creation of file
/home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132927896412.tmp
for container
ERROR XJ001: Java exception:
'/home/lc/projects/webarkivering/scripts/sql/cdxdb/tmp/T1132927896412.tmp (Too many open files):
java.io.FileNotFoundException'.
ij>

Derby creates files in the tmp directory at a rate of about 8 per
second.  If it doesn't close all of these, it would run out of FDs
(ulimit 1024) before long.

I would file a bug report, but db.apache.org isn't responding.

-Lars


Mime
View raw message