db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Duncan Groenewald <duncangroenew...@xtra.co.nz>
Subject Re: AW: Howto import data from different databases
Date Thu, 31 Aug 2006 10:38:00 GMT
I started with a console program and properties file but have recently 
upgraded it to be a swing gui with databases defined in a properties file.

You then select the source database (can be a CSV file), then the 
schema, then the table, then the columns you want. You can also reorder 
the columns to match the target databases columns and visa versa. Then 
you selected the target database, schema, table and columns and click copy.

The program checks the number of source records and that the source and 
target columns match and then starts copying. Null fields are not 
included in the sql statements which are dynamic so may have performance 
problems for very large amounts of data.

Also I found some drivers or something crashed after a large number of 
rows (probably my bad code). Anyway the quick fix was to reconnect (new 
Connection) every 1,000 records and it now seems to run fine. Loading 
from a Notes database via a Notes SQL ODBC driver (odbcjdbc) does about 
100 records per second. I have lots of disparate data but seldom have to 
worry about more than 100,000 records.

The copying runs in a background thread and shows progress and when its 
done it compares number of target records with original.

Code is not pretty but given I often have to do conversions etc. and I 
have total control over the code to be able to handle any strange data 
or conversions its all I need. Send me your email address if you want a 
copy of the code or screen shots. You will have to read the code to 
figure it out !!

All up I have probably spend 20 hours on it.


Peter Neu wrote:
> Ok. I will check that out. I'm also thinking about doing my own
> import/export SW. Point is I don't want to do it manually but configure it
> as a quartz job which updates everything every now and then. 
> How high do you estimate the work which has to be put in this kind of
> software? Basic CRUD Programming I have done for some time now.
> Any idea what shortcuts there are?
> Cheers,
> Pete 
> -----Ursprüngliche Nachricht-----
> Von: Duncan Groenewald [mailto:duncangroenewald@xtra.co.nz] 
> Gesendet: Donnerstag, 31. August 2006 08:18
> An: Derby Discussion
> Betreff: Re: Howto import data from different databases
> I have written my own import/exporter to do just this because I found 
> the commercial import/export applications invariable failed on me 
> somewhere. Try Aqua Data Studio, although I can't recall how well it 
> supports derby.
> Duncan
> Peter Neu wrote:
>> Hello,
>> is it possible to import large amounts of data from different db’s 
>> into derby? I would like insert some tables from a big oracle db into 
>> derby to make
>> it available to small applications which I can then distribute. If 
>> that is possible can I also do synchronization, meaning updating
>> the distributed derby db’s?
>> Cheers,
>> Peter

View raw message