db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kristian Waagan <Kristian.Waa...@Sun.COM>
Subject Re: order by
Date Fri, 03 Mar 2006 13:22:11 GMT
Thomas Vatter wrote:

>The result of distributing application and database over two machines is:
>Import of the critical data now works, no hanging at 1200 records
>(I need to start derby with -Xmx1024m else an outofmemory exception
>is thrown). My import algorithm has to be reviewed, it allocates 
>memory in a loop, 100mb when looping 1300 times. The original ordering 
>problem does not appear at this point.
>Now the less good news. After work, I turned off the computer where 
>derby is installed. I know derby expects a shutdown, but I just 
>pressed CTL-C in its terminal window and made a shutdown for the os. 
>This morning I booted the computer and started derby. I was surprized
>about the memory consumption in the task list. The derby process was
>consuming the complete memory, 1024mb, resident part 30mb (os is
>suse9.3). I tried an import of another spreadsheet with 1220 records 
>in this situation. The import worked. I displayed the newly imported
>data in my application, the data are listed slowly - and the 
>original ordering problem appears. To check if the composite index on
>the sorting columns was corrupted I dropped and recreated the index.
>Same ordering problem. Perhaps the cancelling of the derby server in
>the evening caused the problem, I will redo the whole thing without 
>stopping the server.
>
>  
>

Just to experiment a little, I created a program that imported data into 
Derby.
Only one table with three CHAR(100) columns. I inserted approximately 5 
GB (of text) with the default Derby 10.1 and JVM settings. A count(*) 
reported this:

ij> select count(*) from biginsert;
1
-----------
17476267

1 row selected

Memory usage was low at all times, and as you can see there are over 17 
million rows in the database. So data volume per se should not be a 
problem here, it must be something else causing your trouble (schema 
layout? column types?).

For us to be able to help you, is there any possibility you can give us 
more details about the database schema (tables, columns, types, 
indexes/indices), the data itself and/or the import program?

The fact that you need to start the JVM with -Xmx1024m indicates 
something is wrong. It can be Derby's fault, but it very hard to tell 
without more information.




thanks,
--
Kristian

Mime
View raw message