db-torque-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Monroe <Greg.Mon...@dukece.com>
Subject RE: newbie and generator rewriting
Date Thu, 11 Feb 2010 19:55:54 GMT
More comments in line.. but first, one think to note about 4.0 is that we're 
also planning to convert the XML file format from DTD to XSD.  See:

http://people.apache.org/~gmonroe/torqueSchema/

So when we talk about various tasks needing work, this is part of it.  Oh,
and you might want to read some of the following email threads:

http://www.mail-archive.com/torque-dev@db.apache.org/msg04852.html

http://www.mail-archive.com/torque-dev@db.apache.org/msg04890.html

I think there are some other topics around that same time period as well.

As I said.. more comments in line.

> Thomas Fischer said:
> > > ...
> > > Ah ok, you are using the sql->xml part of the generator ? This is
> indeed an
> > > area where noone else of the torque developers is really interested
> in;
> 
> > May I ask why ? I had guessed that in the past as a user,  but is
> > something I don't understand.
> 
> Forgive me, but personally I think reverse engineering from the database
> is
> an inherent pain. You tend to loose important information, e.g.foreign
> key
> relation (at least using Mysql's MyIsam tables, not so sure for other
> DBs).
> Also, reverse type mapping can be difficult. In simple cases, you can
> guess
> the Java type automatically, but you will never get the Torque types
> booleanchar and booleanint back from the DB. And types might be different
> for every supported database. Getting ranges is difficult(can one
> determine
> that a column is varchar(134) or number(10,2) ? No idea.)
> Then, there's default value, no idea if you get that back. And so on. My
> energy is too limited for this wide field.
> 
> The feeling between the Torque developers was that the reverse engineered
> file serves as starting point but must be tweaked manually afterwards.
> But
> I'm happy to be proven wrong....

I agree TF here. It's hard and will always be wrong for someone or some
version of SQL.  Remember Torque has to work across all major DB server. 
But, as TF said, it's a good tool to get a basic snapshot that can be modified 
to match. IMHO, It's part of the nice rapid application development point of 
view that Torque has.

FWIW, if you dig into how you would do this, you find that you're dependent 
on what the JDBC driver returns about the database. If I remember the JDBC 
specs, the meta data you have to depend on has a lot of "holes" in it as far 
as finding the underlying DB structure.  Plus, there is a lot of variation 
in how different drivers implement it.

Theoretically, I think the SQL standard has definition for a set of system
tables (e.g. SYSCATELOG, etc) that has detailed DB Schema info.  But they
aren't something people strive to keep to standard.  (I think Oracle 
discourages using them...)

That said, there is probably room for improvement here if someone wants 
to dig into it.

> 
> > Is the scenario in which you have to write code for an existing db so
> > unusual?
> 
> For me, it does not happen often. Usually the projects I do is developing
> from scratch.

I've found myself adding Torque objects on top of other people's DB a lot.
In this case, I tend to say "managing the schema" is the other code's job and
Torque's job is just to let my code interact with the data.  In this situation, 
the sql->xml plus a little tweaking can easily produce the "no-sql" objects
I need.  

> 
> > The sql->xml is also useful when you change the db and you want to
> > regenerate the torque classes. How you do that: do you manipulate
> > manually the XML ?
> 
> Yes, and create the alter table script manually. Altering an existing
> database requires good thinking and testing anyway most of the cases, so
> handcoding the SQL is the smallest part.
> 

I think it's important to consider a couple of cases for SQL updates.  The
first is the major structural changes that requires a lot of testing, etc. 
as TF talks about above.

The second is case is the additive / minor updates one.  E.g. I've added more 
tables to  support more features and maybe a few columns to existing tables.  
This is a much more straight forward change and sql-xml is very useful in this.

FWIW, in the latter case, I tend to do is create the SQL build from the 
*schema.xml that has been checked in.  Create a new DB with tables from it.  
Do an sql->xml process on the development/test DB and on the new DB.  Compare 
the two and resolve any "whoops... I added that column to the DB but not the 
XML. Repeat until they match... (plus keep and eye out for missing FK/Indices).

Then I'll use this XML with the DDLUtils API to create the basic ALTER script.
It has the ability to do an compare of the real DB to an XML and create a
nice SQL script.  This isn't perfect, but it's a good start and with some 
review and quick editing, give me the script to run during the code upgrade.

One other benefit of this is that since I run my main app on different 
production machines, it catches / handles the cases where one production server
has been "patched" with a new feature/db setup to meet a client need prior 
to the feature being fully rolled out.  The generated alter script will take
this into account.

I don't think this could ever be fully automated but having the various tools
makes it MUCH more faster to do and less error prone.

> > > we
> > > could use help in that area. In fact, while reorganizing the
> generator
> this
> > > is a part that still needs rewriting (in my opinion, throw out the
> > > ant/texen dependency and create front ends for the different build
> systems.
> > >
> > Different build systems ? What are you thinking about besides ant and
> > maven ?
> 
> Only those two :-) But the old generator is strongly tied to ant which is
> quite pointless for the maven part. It is not a big problem, but the code
> is more complicated and heavy than it has to be.
> In my eyes, the better architecture is to provide the functionality in
> plain java (build-system-agnostic) and provide adapters for the different
> build systems.
> 

Agree... but as a diehard Ant user who only uses Maven if it is required 
(like Torque building and testing...lol)... we should NOT drop Ant 
functionality just for Maven... but I think there is a happy median here
as long as both cases are considered.

> > > Maybe one could also use ddlutils (http://db.apache.org/ddlutils/), I
> did
> > > not check this).
> > I don't know the tool but I have had a look at the web site. In my
> > opinion the integration makes sense if the ddlutils heavily simplify
> the
> > actual code and the tool is widely used (which guarantees is powerful
> > and stable).
> > I see that they last released  it in 2007, quite a long time.
> 
> You are right, there is currently not much traffic there, but I'd guess
> Thomas Dudziak is still around. This was an idea to save double work in
> Torque, but I did not check in detail. For example, I'm not sure that
> ddlutils has maven support... But personally I'd leave that decision to
> the
> person who actually does the work.

I think I was the one who proposed using more of the DDLUtils code in 4.0. 
The thought here was that it does not make sense to have two projects 
doing basically the same thing.  I think we should keep the tasks in 
Torque, but the underlying code should use the DDLUtils API to do the heavy
lifting.

As TF says, there are DDLUtils people around, if we show the need and provide
the code patches to make the two work together, it will get added.  Worst case
is that a Torque committer could make a request to be allowed to commit to the
DDLUtils project (I think we have the rights already, but asking is the proper
way) or maybe request to "adopt" the project if the "owner" has moved on.

Hope this isn't too confusing since I've been typing on it piece meal over
the day.

Greg
DukeCE Privacy Statement:
Please be advised that this e-mail and any files transmitted with
it are confidential communication or may otherwise be privileged or
confidential and are intended solely for the individual or entity
to whom they are addressed. If you are not the intended recipient
you may not rely on the contents of this email or any attachments,
and we ask that you please not read, copy or retransmit this
communication, but reply to the sender and destroy the email, its
contents, and all copies thereof immediately. Any unauthorized
dissemination, distribution or copying of this communication is
strictly prohibited.

---------------------------------------------------------------------
To unsubscribe, e-mail: torque-dev-unsubscribe@db.apache.org
For additional commands, e-mail: torque-dev-help@db.apache.org


Mime
View raw message