db-derby-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Segel" <mse...@segel.com>
Subject RE: How to monitor progress during data import
Date Fri, 28 Jul 2006 16:15:49 GMT
Uhm, well, there's an ugly kludge that might work...

You could always count the number of lines in the input file before you
issue the "load" command. This will tell you the total number of rows that
"should" be loaded.

You could then periodically monitor the number of rows in the table and this
should give you a "rough" percentage of completion.

Note that in theory it should work, just not work well. Rather than doing a
row count, can't you look in a sys table to find the number of rows in a
given table? Also, if the "load" command locks the table in exclusive mode,
will it also lock the sys table's entry as well? Or will the table be
updated in real time, or only after each commit.

Please also note that I am talking in generalities. Since you're talking
about using Java, then you're probably going to consider a multi-platform
solution.

HTH

-G


> -----Original Message-----
> From: Ryan Bobko [mailto:ryan@ostrich-emulators.com]
> Sent: Friday, July 28, 2006 8:59 AM
> To: Derby Discussion; 'Fantry, John'; 'Derby Discussion';
> 'mikem_app@sbcglobal.net'
> Subject: RE: How to monitor progress during data import
> 
> If you don't have access to the input stream, then this method won't
> help too much. However, if you do, Java 1.4.2 has a ProgressMonitor
> class that can pop open after a specified amount of time with a progress
> bar. It also has a ProgressMonitorInputStream which can wraps the input
> stream.
> 
> ry
> 
> > John
> > There are two ways to attack this problem. Both ways use Swing and the
> > JPogressBar. The easiest way is to use
> > the 'setIndeterminate(true) method and you will get a little bar that
> moves
> > back and forth while the file is
> > being imported. Set it up in a separate thread and when it the import is
> > complete the thread will end and use
> > setIndeterminate(false) and it will stop. I use this for both the
> backup and
> > restore funcions for Derby. Another way
> > is to count the number of bytes coming in as you read your text file.
> I have
> > the code to do this at home, which I
> > can give you tomorrow, but I suggest the first method. Your user doesn't
> > really need to know how much has been
> > copied, but only that something is being done. If the procedure will
> take
> > more than five minutes then I could see
> > the need. But if you need to do it, I will give you a way to approach
> this
> > issue.
> >
> > Dan
> >
> > -----Original Message-----
> > From: Fantry, John [mailto:JFantry@analogic.com]
> > Sent: Wednesday, July 26, 2006 9:00 AM
> > To: Derby Discussion; mikem_app@sbcglobal.net
> > Subject: RE: How to monitor progress during data import
> >
> >
> >
> > I am importing the data using the built in system procedure
> > SYSCS_UTIL.SYSCS_IMPORT_TABLE.  This procedure takes the name of the
> file to
> > import as an argument.  Derby opens the file using whatever mechanism it
> > wants.  I don't have an opportunity to wrap the stream myself.
> >
> > -----Original Message-----
> > From: Mike Matrigali [mailto:mikem_app@sbcglobal.net]
> >
> > Sent: Wednesday, July 26, 2006 11:27 AM
> > To: Derby Discussion
> > Subject: Re: How to monitor progress during data import
> >
> >
> >
> > Andreas Korneliussen wrote:
> > > Fantry, John wrote:
> > >
> >
> > >> I am using Derby to import a large text file into a table.  I would
> >
> > >> like to be able to display a progress dialog to the user while the
> >
> > >> import is processing.  In order to provide a useful progress bar I
> >
> > >> need to be able to calculate the % complete of the import procedure.
> >
> > >> Does anyone know if this can be done?  The table is locked during the
> >
> > >> import so I can't execute any SELECT COUNT(*) statements in a
> > separate
> >
> > >> thread.  Perhaps Derby outputs some statistics to a log file?  I did
> >
> > >> stumble across the RunTimeStatistics attribute in the Derby docs, but
> >
> > >> this attribute is only meaningful after the import is complete.  I
> >
> > >> need insight into the import procedure while it's executing.  Any
> > ideas?
> > >>
> >
> > >
> >
> > >
> >
> > > Have you tried running the SELECT COUNT (*) statement using
> > transaction
> >
> > > isolation level "read-uncommitted " ?
> > Do note that derby must scan all the rows in the table to execute
> >
> > count(*), so checking on the progress using count(*) is a significant
> > overhead depending on the size of the table.
> >
> > For your application it may be possible to layer a stream on top of
> the file
> > which could tell your app how much has been read while otherwise just
> > passing the bytes from the file to derby.
> > >
> >
> > > Regards
> > > Andreas
> > >
> >
> > >
> >
> > >
> >
> >
> >
> > ****************************************************************
> > The information transmitted in this message is confidential and may be
> > privileged.  Any review, retransmission, dissemination, or other use
> of this
> > information by persons or entities other than the intended recipient is
> > prohibited.  If you are not the intended recipient, please notify
> Analogic
> > Corporation immediately - by replying to this message or by sending an
> email
> > to DeliveryErrors@analogic.com - and destroy all copies of this
> information,
> > including any attachments, without reading or disclosing them.
> >
> > Thank you.
> >
> >
> 




Mime
View raw message