incubator-bloodhound-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jure Zitnik <j...@digiverse.si>
Subject Re: [Apache Bloodhound] #404: Populate default schema on product addition
Date Tue, 26 Feb 2013 14:16:51 GMT
On 2/26/13 10:01 AM, Olemis Lang wrote:
> On 2/25/13, Jure Zitnik <jure@digiverse.si> wrote:
>> On 2/24/13 7:07 AM, Olemis Lang wrote:
>>> On 2/22/13, Jure Zitnik <jure@digiverse.si> wrote:
> [....]
>
>> Let's say that the 'environment_created' method of the component does
>> the following:
>> 1. creates table named 'my_table'
>> 2. inserts into 'my_table' data that aggregates/references some data
>> from system tables ('ticket', 'component',...)
>> 3. creates a file in 'env.path' that holds some environment specific data
>> 4. inserts it's database version into 'system' table
>>
>>
>> Now, let's discuss the relevant scenarios: (global) environment creation
>> (install from scratch), product addition and environment upgrade.
> There's an important case : upgrade with installed plugins and
> preexisting data . maybe covered by the third one . It involves data
> migration and a few other corner cases ... but your initial scenarios
> are just find to get this sorted out .

I started upgrade scenario discussion in a separate thread last week, 
subject '[BEP-0003] Custom (3rd party plugin) table upgrade to 
multi-product'.

>> I'll
>> describe scenarios for both cases, first for non multi-product aware
>> component and then for multi-product aware component.
>>
> I'll follow this sequence with the purpose in mind of finding a way to
> work around the current design limitation of having O(p) plugin tables
> .
>
> So I'm basically thinking of having one of the following DB schemas :
>
>    1. A single table scoping all plugin resources at product level
>    2. Two tables : one unchanged for products in global scope ,
>        the second one for product resources .

jftr, I understand reasoning for trying to find out another way and not 
go with the O(p) plugin tables. Please note that the current approach 
has been proposed in BEP-0003 a couple of months ago and that all work 
done till now was following that design. The option of adding 'product' 
column to 3rd party plugin tables (the same way as for system tables) 
was assessed when preparing the BEP-0003 database translation proposal. 
It was dropped for the reasons stated below.

The proposed 2 options for the database schema are not fundamentally 
different. We could go with single table by adding the 'product' column 
and scoping the table the same way as we do with the system resource 
tables. I believe that's (1) in your list. In both scenarios we run into 
the same obstacles as explained below.

Again, IMO that is not doable mainly because we don't know 
(semantically) what's in those tables. We don't know the schema, we 
don't know what data it could aggregate and/or reference, we don't know 
what changes to the schema and data could be performed during the 
upgrade etc.


> I'll only care about DB stuff because that's actually what really
> matters and the cause of this bottleneck .

I'd say that the DB stuff matters, but we can't just ignore other issues.

>   I'll also ignore MP-aware
> plugins / components given the fact that there is no such thing yet
> and we will be able to do whatever we consider necessary to make that
> work , as opposite to legacy plugins .
+1

>> 1. Global environment creation
>> -- Non multi-product aware component
>> In this case, the components
>> 'IEnvironmetSetupParticipant.environment_created' should be executed in
>> global environment only (as there are no products). Result of this would
>> be 'my_table' table with aggregated/referenced data from 'null' product
>> scope (global scope). Note that the translator does not prefix table
>> names when product is 'null'. File that the component creates would be
>> stored in the global environment path, the 'system' table would get a
>> record with 'null' product prefix.
>>
>> [...]
> Looking at this with the previous goals in mind , IMO at this point we
> should be doing this (numbers will match scenarios listed above)
>
>    1. create plugin table with extra 'product' column for prefix
>        and perform all other outstanding DDL and DML statements
>        based on this modified schema .
>    2. create plugin table with extra 'product' prefix column

How would we be handling indexes, unique keys and other possible 
constraint, etc. in any of the above cases? Automatically adding 
'product' column to unique constraints? I don't believe there's a 
general rule that we could apply in regards to (but not limited to) 
constraints...

> [...]
>
>> 2. Product (with 'MYPRODUCT' prefix) addition
>> -- Non multi-product aware component
>> Adding a product should invoke component's
>> 'IEnvironmentSetupParticipant.environment_created' method
> Ok . We agree up to this point , afaict since the beginning .
>
+1

>> from within
>> 'ProductEnvironment('MYPRODUCT')'.
> Nope . I disagree here . IMO , this should be happeing immediately
> after creating the product either in MP system's CRUD handler , a
> dedicated implementation of a product resource listener or a
> combination of both .
>
I understand this, though I don't agree for the reasons I stated above. 
In my opinion, and I think Andrej also mentioned this in his reply, from 
our perspective what happens within component's 
'IEnvironmentSetupParticipant' method is a black box. We can do the SQL 
translation through our proxy but I'm strongly against recording SQLs 
and doing replays.

For example, how would MP system (or whoever we choose to do the 
replays) replay the following really simple SQL, per-product:
SELECT * FROM component WHERE name='Product A component';
assuming that 'Product A component' is only defined in product 'A' ...

>> This would enable SQL translator in
>> that scope with the known rules. Result of this would be
>> 'MYPRODUCT_my_table' with aggregated/referenced data from 'MYPRODUCT's
>> scope. File would be stored in 'MYPRODUCT' product's path. 'system'
>> table would get a record with 'MYPRODUCT' product prefix.
>>
> In both scenarios I mentioned above (1) and (2) we'll need the
> translators , that's a fact.
>
> Nevertheless notice that global upgrade would shape the schema for any
> product scope as well . So my idea is to shut off DDL statements at
> this point . OTOH DML statements will remain translated considering
> product prefix as data still needs to be inserted / updated / selected
> / removed .

As much as I would like to, but i really don't think the 'would shape 
the schema' would work for reasons stated above (constraints etc.).

I don't understand what you meant with 'shut off DDL statements'? If 
nothing else, all DDLs would need to be translated in such a way to take 
into account the 'product' column.

> [...]
>> 3. Environment upgrade
>> -- Non multi-product aware component
>> Following the normal process (in 'trac.env.Environment'), each non
>> multi-product setup participant should be upgraded (if required) from
>> all defined product scopes. This would leverage
>> 'IEnvironmentSetupParticipant.environment_needs_upgrade' and
>> 'IEnvironmentSetupParticipant.upgrade_environment', executing those two
>> from within all product scopes (+global one).
> JFTR : this is part of what I referred before as «replay» as I was
> actually thinking of an upgrade with products in the DB , preexisting
> plugin data and pending upgrades applied upon products in a batch
> while still inside the global upgrade loop . IMO this should happen
> inside the global upgrade loop .

Upgrade needs to be discussed, I agree, that's why I started a 
discussion last week.

> [...]
>> This is how, in my understanding, things should be. I'm not sure there's
>> really any other way of implementing this ...
>>
> We shall see what you all think about my proposed hack . I look
> forward to read further comments .
>

My proposition would still be to go with the O(p) solution. We might 
think of making translator a bit more configurable as Andrej suggested.

Cheers,
Jure


Mime
View raw message