openjpa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Teresa Kan" <tck...@gmail.com>
Subject Re: [jira] Updated: (OPENJPA-464) Performance improvement with Statement Batching support
Date Tue, 18 Dec 2007 20:18:32 GMT
Hi Pinaki, I embedded my comment in your question. See <teresa>...</teresa>

On Dec 18, 2007 1:30 PM, Pinaki Poddar <ppoddar@bea.com> wrote:

> Hi Teresa,
>  > Our internal design was in the extension of the OperationUpdateManager
>  Will you please clarify what is meant by 'Our internal design'?

  >> <teresa> It was the product that I worked with had the statement
batching support with OPENJPA version 1.0.x . </teresa>

>
>
>  > I did this before we can't change the code in the DBDictionary at the
> first place.
>  Not clear what you mean. The patch does modify DBDictionary.

>> <teresa>Because we treated it as proprietary, so we can't update the code
and configuration in the DBDictionary. </teresa>

>
>
>  UpdateManager interface is used to customize the way that updates are
> made to database records.
>  UpdateManager works in conjunction with PreparedStatementManager which
> manages prepared statement execution.
>  So as you are supporting batching of SQL, a extended
> PreparedStatementManager is the correct way to go which is configured via
> UpdateManager.
>  Whereas, your patch modifies PreparedStatementManagerImpl to accomodate
> batching logic. I do not think this is in-line with original designer's
> view. Otherwise AbstractUpdateManager will not have a
> newPreparedStatementManager() method.

>> <teresa> I understood that the PreparedStatemetnManager is associated
with the UpdateManager. However, the batch support really depends on the
data store which supports it or not. Even though you set the batchLimit on
the updateManager, if the database does not support it, it does not make
sense to set it there. That's why I moved the code from new
PreparedStatementManager and UpdateManager to the existing
PreparedStatementManagerImpl.
For my existing code, I just removed my new PreparedStatementManagerImpl and
changes in my extended OperationOrderUpdateManager to delegate the code to
the existing PreparedStatementManagerImpl. I still can support our
proprietary batching configuration which delegates to the new
implementation.
</teresa>

>
>
> > Therefore, there is no overhead for the non-batch process - the
> implementation is separated in a way that you don't need to use the plugin
> class.
>
> I am not making any efficiency/performance argument against your patch. I
> am making a design argument. And I am in favor of a plug-in UpdateManager
> that supports batching alongside the existing one that does not. That is
> in-line with plug-in architecture of OpenJPA for feature extension.

<teresa>I like to pool more people's opinion on this.. Since most of
databases support batching, I don't think we need a plugin architecture for
this feature. Unless there is only one database using this feature, then I
agree with you that  the plug-in architecture is the way to go.
</teresa>

>
>
>
>
> -----Original Message-----
> From: Teresa Kan [mailto:tckan1@gmail.com]
> Sent: Tuesday, December 18, 2007 11:06 AM
> To: dev@openjpa.apache.org
> Subject: Re: [jira] Updated: (OPENJPA-464) Performance improvement with
> Statement Batching support
>
> Hi Pinaki,
> Our internal design was in the extension of the OperationUpdateManager, so
> the batch and non-batch support was separated. I did this before we can't
> change the code in the DBDictionary at the first place. When I moved the
> function from our internal implementation to the openjpa layer, I think to
> enable the batch from DBDictionary is a better decision. From usability
> point of view, it is better to configure the batchLimit under the
> DBDictionary instead of UpdateManager. My implementation tested the
> batchLimit at the constructor of PreparedStatementManagerImpl , if the
> Dictionary does not support batch, then it will be disabled and the
> non-batch process is used instead. The MAP in this class will not be
> initiatized until it is necessary to use. Therefore, there is no overhead
> for the non-batch process - the implementation is separated in a way that
> you don't need to use the plugin class. By default, most of DB Dictionaries
> are disabled with the exception of DB2 and Oracle.
>
> Teresa
>
> On Dec 18, 2007 12:36 PM, Pinaki Poddar <ppoddar@bea.com> wrote:
>
> > Hi Teresa,
> >
> >  Batching support should be added to a separate implementation of
> > PreparedStatementManager rather than existing
> > PreparedStatementManagerImpl trying to act on both batch and non-batch
> > mode. A new BatchPreparedStatementManager can be plugged-in by
> > overwriting AbstractUpdateManager.newPreparedStatementManager().
> >
> > That way, one will not require a Map in
> > BatchingPreparedStatementManager but only a instance variable e.g.
> 'PreparedStatement _sql'.
> > The flushInternal() checks whether the current SQL requires a new
> > PreparedStatement or the existing one will do. If it requires a new
> > PreparedStatement then existing _sql is executed + _sql is reassigned
> > the current one. Otherwise, _sql.addBatch(current).
> >
> > This design will keep batch/non-batch support separate and will help
> > maintenance in future.
> >
> >
> >
> >
> > -----Original Message-----
> > From: Teresa Kan [mailto:tckan1@gmail.com]
> > Sent: Tuesday, December 18, 2007 9:30 AM
> > To: dev@openjpa.apache.org
> > Subject: Re: [jira] Updated: (OPENJPA-464) Performance improvement
> > with Statement Batching support
> >
> > I would like to hear any comment from the community and would like to
> > commit the changes before the holidays. If there is no objection by
> > end of tomorrow. The patch will be committed.
> >
> > Thanks and happy holiday!
> > Teresa
> >
> > On Dec 13, 2007 4:15 PM, Teresa Kan (JIRA) <jira@apache.org> wrote:
> >
> > >
> > >     [
> > > https://issues.apache.org/jira/browse/OPENJPA-464?page=com.atlassian
> > > .j ira.plugin.system.issuetabpanels:all-tabpanel]
> > >
> > > Teresa Kan updated OPENJPA-464:
> > > -------------------------------
> > >
> > >    Attachment: OPENJPA-464.patch
> > >
> > > Attach the patch including the OPENJPA documentation changes.
> > >
> > > > Performance improvement with Statement Batching support
> > > > -------------------------------------------------------
> > > >
> > > >                 Key: OPENJPA-464
> > > >                 URL:
> https://issues.apache.org/jira/browse/OPENJPA-464
> > > >             Project: OpenJPA
> > > >          Issue Type: Bug
> > > >          Components: kernel
> > > >    Affects Versions: 1.1.0
> > > >            Reporter: Teresa Kan
> > > >             Fix For: 1.1.0
> > > >
> > > >         Attachments: OPENJPA-464.patch, statement batch
> > > > design_1211.doc
> > > >
> > > >
> > > > The current OpenJPA implementation did not provide the SQL
> > > > statement
> > > batching. All SQL statements will be executed one statement at a
> > > time to the database. Consequently, the runtime performance was
> > > decreased due to lots of database flows. JDBC Specification provides
> > > the batch capability for insert, update and delete statements
> > > through the
> > addBatch() and executeBatch() APIs.
> > > We should be able to take advantage of this capability to support
> > > SQL statement batching in OpenJPA.
> > > > According to the old version of the OpenJPA manual (i.e., Kodo),
> > > statement batching was part of the initial functions. Conscious
> > > decision by BEA that this function was not contributed back to
> > > OpenJPA. We can still use this info as the implementation base with
> > > some
> > modifications.
> > > > I have completed the work for this statement batching support and
> > > > the
> > > patch has been tested by CTS against Derby and DB2, OPENJPA
> > > regression test as well as our internal FVT test bucket.  The
> > > following section describes the design and implementation info. I
> > > also attached the whole design documentation and the patch in this
> > > jira. Once the design and implementation are accepted, then I will
> > > update the OPENJPA manual to include this function. Thanks,
> > > > Design and implementation:
> > > > *     Configuration:
> > > > o     Batch Limit value:
> > > > §     0 - Disable batch support.
> > > > §     -1 - Unlimited number of statements for a batch.
> > > > §     Any positive number - Maximum number of statements for a
> batch.
> > > > o     By default, the batch support is based on each Dictionary to
> > > define the default batch limit. Currently only DB2 and Oracle
> > > dictionaries are set the default batch limit to 100. The default
> > > batch limit for rest of the dictionaries is set to zero (disabled).
> > > > o     To enable the batch support, user can specify the following
> > > property in the persistence.xml file:
> > > > <property name="openjpa.jdbc.DBDictionary" value="BatchLimit=25"/>
> > > > or <property name="openjpa.jdbc.DBDictionary"
> > > > value="db2(batchLimit=25)"/>
> > > > *     Basic design is to cache all the insert/update/delete
> statements
> > > during the execution of the
> > > PreparedStatementManagerImpl.flushInternal()
> > > method. There is a cache structure which uses the LinkHashMap to
> > > maintain the order of the SQL statements for execution:
> > > > o     _cacheSql - a LinkHashMap to store the rows that associate
> with
> > > one PrepareStatement. Key: SQL statement string; Value: array list
> > > of
> > rows.
> > > > During the PreparedStatementManagerImpl.flush() process, it will
> > > > go
> > > through the cache to prepare the SQL statement; add the statement to
> > > the batch; and execute the batch when the batch limit is reached or
> > > all the rows are processed for that statement. Validate the update
> > > count after the
> > > executeBatch() method.
> > > > *     If the batch limit =0 (disabled), execute the statement as the
> > > normal process; no need to use the batching process.  Same rule
> > > applies to the statement that only has one row, execute it as the
> > > normal
> > process.
> > > > *     The batch process will be disabled if the primary key
> generation
> > > is used the Identity strategy. When the GeneratedType=IDENTITY, we
> > > need to get the ID value right away for the in-memory entity to use.
> > > Therefore, we can't batch this kind of statement.
> > > > *     Batch exception process: a checkUpdateCount() is used to
> > validate
> > > the batch process after the executeBatch(). According to the
> > > javadoc, there are three cases to consider:
> > > > o     Case of EXECUTE_FAILED: (-3):
> > > > §     This is a failure case. If the action is UPDATE or there is
> > > FailedObject, treats it as OptimisticException. Otherwise, throws
> > > the SQLException.
> > > > §     This is the same process as current implementation.
> > > > o     Case of  SUCCESS_NO_INFO: (-2):
> > > > §     We treat this as successful case and log the info in the log.
> > > > o     Case of 0:
> > > > §     If there is a FailedObject or the action is INSERT, then
> throws
> > > the SQLException. Otherwise, treats it as successful case.
> > >
> > > --
> > > This message is automatically generated by JIRA.
> > > -
> > > You can reply to this email to add a comment to the issue online.
> > >
> > >
> >
> > Notice:  This email message, together with any attachments, may
> > contain information  of  BEA Systems,  Inc.,  its subsidiaries  and
> > affiliated entities,  that may be confidential,  proprietary,
> > copyrighted  and/or legally privileged, and is intended solely for the
> > use of the individual or entity named in this message. If you are not
> > the intended recipient, and have received this message in error,
> > please immediately return this by email and then delete it.
> >
>
> Notice:  This email message, together with any attachments, may contain
> information  of  BEA Systems,  Inc.,  its subsidiaries  and  affiliated
> entities,  that may be confidential,  proprietary,  copyrighted  and/or
> legally privileged, and is intended solely for the use of the individual or
> entity named in this message. If you are not the intended recipient, and
> have received this message in error, please immediately return this by email
> and then delete it.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message