db-torque-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Fischer <tfisc...@apache.org>
Subject Re: Torque features I've added in a forked tree... Merge into main?
Date Mon, 22 Aug 2005 21:17:58 GMT
Hi,

Thanks for suggesting your contribution. I've added comments below. Note
that they are my own personal view on the features you describe, and that
I'd welcome some more opinions on this.

As the subject of this is development of Torque and not using it I'd like
to move the discussion to the dev list, if that is not inconvenient to
you.

On Thu, 18 Aug 2005, Kevin Burton wrote:

> OK.
>
> I've been working on a forked version of Torque for a while now.
> Probably around two years.  I didn't intend for it to get THIS bad but
> it has.  The main problem was I was sooo busy working on my main app
> that I never had a chance to contribute back the changes.

In principle, I think this is a very good idea. However, a warning at the
beginning: As two years is quite a long time, you might have noticed that
the structure of Torque was changed quite a bit in the meantime. I guess
you branched before the generator was decoupled. Merging will not be easy.

>
> Anyway.  I'm starting to work on some new projects and figured this
> was a good enough time as any to correct this problem.
>
> I'm going to try to dig into the Torque sources to see about the
> current status but I thought I'd post a list of the features I've
> added.  I still have to review my code but I think this is pretty much
> everything.
>
> ^U prejoin support and the prejoin API.  This allows you to fetch all
> the object references for a list of objects at once.  For MySQL (and I
> assume other databases) this has a significant performance advantage.
> Every SQL call costs about 1ms so for a list of 1000 items this can
> save 999ms.

I'd like to know more about this. What exactly does it do ? How is it 
implemented ?

>  support for prepared statements (which are MUCH faster in MySQL)

Torque is using prepared statements for updates and deletes, and there is 
some support for prepared statements on selects also. However, the support 
of prepared statements for selects is quite limited at the moment, so if 
you can do better, why not ? But merging this will be a real mess.

>  when performing an UPDATE/INSERT only send columns that have been 
> modified

Hm. Not sure of that. Your data could get a mess if someone else had 
modified the same object in the meantime, and I do not see that the 
performance gain is large enough to justify that. It could even decrease 
performance, if you tell dbcp to cache the prepared statements, and you 
have to use another prepared statement each time because you have modified 
other columns. Maybe if some people are interested, one could make it 
configurable via a generator option. This would also solve the problem of 
backward compatibility.

>  client-side join for IN clauses by specifying a List of BaseObjects
> which have their getId() method called.

This sounds like a rather specialized thing to me. It sounds as if it 
would only work on primary keys. Is that correct ?

>  lazy result set processing.  For example if you have a table with
> 10000 columns and you only call getColumnA() then we don't have to
> fetch the remaining columns.  This has a HUGE performance advantage at
> runtime.

I'd like to know more about this. I would guess that the object would have 
to keep the database connection open either until all of its getters are 
called or until its destructor is called. Is that correct ?

>  New database connection pool called BDCP which is simple to 
> debug/understand, FAST, and reliable.  We had countless problems with 
> DBCP.

I am against having a db connection pool inside Torque. Reason is that a 
db connection pool is a tool which is too general to be embedded in a 
project like Torque. It should be a standalone project.

Don't get me wrong. I have no problem at all to include an adapter for 
another pool besides dbcp into Torque.

Which dbcp version are you talking about ? From what I have heard 
so far, people are quite content with dbcp 1.2.1. Only issue I know of 
which is open is that there are problems with a serializable checked 
non-auto-commit connections.

>  save() should update the identity of the object with ID that was 
> generated on the server.

Not sure if I grasp what you mean by this. You mean if an id is 
autoincrement, and you save a new object, then the id should be updated in 
memory ? This is already the case. But maybe I misunderstood you. Please 
correct me if I'm wrong.

>  BETA support for memcached. This means you can call retrieveById and 
> the prejoin API and have the results come from memcache and NOT from a 
> SQL box.  This should theoretically be much faster but I'm still working 
> on it.  The results were actually 4x slower but I'm optimistic that I 
> can have this fixed.

There is an experimental interface for using JCS as memory cache in Torque 
right now. You have to set the generator option useManagers=true for it. I 
have no big experience in this part of Torque, maybe someone else who is 
using the managers should comment on it.

>  REPLACE support for MySQL

As a non-mysql user, what does REPLACE do ? Is this standard SQL ? Does it 
also work on other databases ?

>  Support for working in MASTER/SLAVE environments when working 
> with MySQL.  save() and doUpdate, doInsert, etc work on a dedicated 
> connection to the master while SELECTs perform on another dedicated 
> connection. The SLAVE connection could also be used with a load balancer 
> if necessary.

Again, is this mysql-specific ?

>  doSelectFirstRecord which returns the object type instead of just 
> Object when you know there will only ever be one result.  This is just 
> syntactic sugar and prevents casting.

Personally, I am undecided on that. I do not mind casting too much, and it 
adds still another method to the Peer class, making them even more 
stuffed. On the other hand, it might come in handy for some users. 
Then again, it is not difficult to implement this in the templates, so if 
someone has a real need for it, no problem to implement it. Anyone else 
cares to comment ?

>  Support for a HashList mechanism where I can take a left join and make 
> it into a Map where the keys are the left most column and the values are 
> a list of all the items in that key.  This really helps for tree 
> structures since SQL can't return hierarchical structures. This 
> mechansim only supports on level of nesting right now.

I do not understand this right now. Can you give an example ?

>  doSelectAsHashMap which can take a column, 
> and a query, and return the result as a HashMap instead of just a 
> regular List.
>

Same comment as on doSelectFirstRecord. Might come in handy some time, but 
adds yet another method. What happens if the column is not the promary 
key, so that you might have multiple objects for one key ?

As a final remark, some of your features sound really interesting. 
Especially the ability to fetch a whole tree of objects at one time is a 
feature which is certainly often needed but only provided in a limited 
extent by Torque (There are the doSelectJoinXXX methods, but only for one 
level of nesting).

    Thomas

Mime
View raw message