stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirmal Fernando <nirmal070...@gmail.com>
Subject Re: [Discuss] Change Artifacts Stored in Registry to a Readable Format to Support Migrations
Date Wed, 13 Aug 2014 04:19:13 GMT
Re-igniting this discussion, I think registry based approach is not a
viable solution, as what we store here is pretty complex set of
relationships.

IMO we should come up with a DB design and uses a DB, to store these
information.


On Fri, Jun 20, 2014 at 11:04 AM, Isuru Haththotuwa <isuruh@apache.org>
wrote:

> AFAIU writing/reading to/from a shared DB is very convenient when you use
> the registry. We do not need to maintain separate database scripts, etc and
> worry about migration. In case of a complex data structure like the
> topology, the DB schema can be changing rapidly and a nightmare to
> maintain.
>
> If I understand correctly, the problem we have here is storing the data in
> the binary format. Registry provides couple of methods to store them in
> non-binary format, either using RXT or simple name-value pairs. The
> advantage I see in a database is the ability to query and retrieve
> information. If there is a proper way to handle this in registry, maybe by
> using associations/dependencies [1] (We need to check with a registry
> expert), IMHO using registry for persisting this data would be very
> convenient than using a database.
>
> [1].
> https://docs.wso2.org/display/Governance460/Managing+Relationships+of+a+Resource
>
>
> On Fri, Jun 20, 2014 at 10:37 AM, Imesh Gunaratne <imesh@apache.org>
> wrote:
>
>> +1 for direct database approach.
>>
>> Few points:
>> - Registry store resource content as blobs even though the content format
>> is text.
>> - Therefore as Udara has pointed out we will need to write a registry
>> client to do the migrations if we go with the registry.
>> - If we use a set of databases (SM, AS, CC), we will be able to provide
>> database scripts for migrations.
>> - Which I think the most convenient way to migrate data.
>> - However with this approach we will need to use transactions when
>> writing to the database to make sure it works fine in a distributed
>> environment (if SM, AS, CC are clustered).
>>
>> WDYT?
>>
>>
>> On Thu, Jun 19, 2014 at 8:03 PM, Isuru Haththotuwa <isuruh@apache.org>
>> wrote:
>>
>>>
>>>
>>>
>>> On Thu, Jun 19, 2014 at 12:35 PM, Nirmal Fernando <
>>> nirmal070125@gmail.com> wrote:
>>>
>>>> I think we should go for databases.
>>>>
>>> Else, a non binary method supported in registry, such as rxt or simple
>>> name-value pairs.
>>>
>>>>
>>>>
>>>> On Thu, Jun 19, 2014 at 12:26 PM, Udara Liyanage <udara@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> In json also if there is a it will assign the default value for non
>>>>> existing variables when converting from json to java object structure.
>>>>> Still if there is a considerable change in object structure, we have
>>>>> to perform additional work in migration.
>>>>>
>>>>> Say we have introduced an variable x, which links to y in another
>>>>> class so on, then we have to do additional work, otherwise it is in a
>>>>> inconsistent state.
>>>>>
>>>>>
>>>>>  On Thu, Jun 19, 2014 at 12:20 PM, chris snow <chsnow123@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Udara, I'm not sure of the situation with JSON, but when using
XML
>>>>>> it is possible to evolve a schema as long as changes are done in
a
>>>>>> backward compatible way.  For example, if you add an optional field,
>>>>>> the parsing code will be able to read xml created with and without
the
>>>>>> field.  However, IIRC java object serialisation is much more rigid
and
>>>>>> this won't work.
>>>>>>
>>>>>> On Thu, Jun 19, 2014 at 6:45 AM, Udara Liyanage <udara@wso2.com>
>>>>>> wrote:
>>>>>> > Hi Imesh/Dinesh,
>>>>>> >
>>>>>> > Though we used a readable json/xml/text still we can't migrate
>>>>>> seamlessly?
>>>>>> > When migrating we have to read the old json and convert it it
the
>>>>>> new object
>>>>>> > structure.
>>>>>> > Could you please explain how making it readable helps to migrate
>>>>>> seamlessly.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Jun 18, 2014 at 2:19 PM, Imesh Gunaratne <imesh@apache.org>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Hi Dinesh,
>>>>>> >>
>>>>>> >> Great! Please provide your thoughts on the changes required
in
>>>>>> registry
>>>>>> >> persistence logic as you progress.
>>>>>> >>
>>>>>> >> Thanks
>>>>>> >>
>>>>>> >>
>>>>>> >> On Wed, Jun 18, 2014 at 12:27 PM, Dinesh Bandara <dineshb@wso2.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Hi,
>>>>>> >>>
>>>>>> >>> When I started work on [1] and I thought to persist
cartridge
>>>>>> >>> configuration in JSON format in Stratos Manager's registry
and
>>>>>> observed the
>>>>>> >>> above behavior which does not provide the readability
of existing
>>>>>> artifacts.
>>>>>> >>> Will work on [2]
>>>>>> >>>
>>>>>> >>> [1] https://issues.apache.org/jira/browse/STRATOS-568
>>>>>> >>> [2] https://issues.apache.org/jira/browse/STRATOS-664
>>>>>> >>>
>>>>>> >>> Thanks
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Wed, Jun 4, 2014 at 10:18 AM, Imesh Gunaratne <
>>>>>> imesh@apache.org>
>>>>>> >>> wrote:
>>>>>> >>>>
>>>>>> >>>> Hi All,
>>>>>> >>>>
>>>>>> >>>> In Stratos 4.0.0 Stratos Manager, Cloud Controller
and
>>>>>> Autoscaler store
>>>>>> >>>> their artifacts in registry in binary format (Java
objects are
>>>>>> serialized
>>>>>> >>>> and stored). This might cause problems when migrating
an
>>>>>> existing Stratos
>>>>>> >>>> deployment to a newer version with changes in above
artifacts.
>>>>>> >>>>
>>>>>> >>>> Therefore it would be better if we could change
this format to
>>>>>> JSON or
>>>>>> >>>> something similar which could be easily read and
updated if the
>>>>>> definitions
>>>>>> >>>> of the artifacts change in a newer Stratos version.
>>>>>> >>>>
>>>>>> >>>> More importantly we might need to create tasks in
JIRA to prepare
>>>>>> >>>> migration scripts if we do any modifications to
the above
>>>>>> artifacts once
>>>>>> >>>> 4.0.0 release is done.
>>>>>> >>>>
>>>>>> >>>> https://issues.apache.org/jira/browse/STRATOS-664
>>>>>> >>>>
>>>>>> >>>> Thanks
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> --
>>>>>> >>>> Imesh Gunaratne
>>>>>> >>>>
>>>>>> >>>> Technical Lead, WSO2
>>>>>> >>>> Committer & PPMC Member, Apache Stratos
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>> Dinesh Bandara
>>>>>> >>> Software Engineer
>>>>>> >>> WSO2 Inc.; http://wso2.com
>>>>>> >>> lean.enterprise.middleware
>>>>>> >>>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> --
>>>>>> >> Imesh Gunaratne
>>>>>> >>
>>>>>> >> Technical Lead, WSO2
>>>>>> >> Committer & PPMC Member, Apache Stratos
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> >
>>>>>> > Udara Liyanage
>>>>>> > Software Engineer
>>>>>> > WSO2, Inc.: http://wso2.com
>>>>>> > lean. enterprise. middleware
>>>>>> >
>>>>>> > web: http://udaraliyanage.wordpress.com
>>>>>> > phone: +94 71 443 6897
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Check out my professional profile and connect with me on LinkedIn.
>>>>>> http://lnkd.in/cw5k69
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Udara Liyanage
>>>>> Software Engineer
>>>>> WSO2, Inc.: http://wso2.com
>>>>> lean. enterprise. middleware
>>>>>
>>>>> web: http://udaraliyanage.wordpress.com
>>>>> phone: +94 71 443 6897
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Nirmal
>>>>
>>>> Nirmal Fernando.
>>>> PPMC Member & Committer of Apache Stratos,
>>>> Senior Software Engineer, WSO2 Inc.
>>>>
>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>
>>>
>>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PPMC Member, Apache Stratos
>>
>
>


-- 
Best Regards,
Nirmal

Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.

Blog: http://nirmalfdo.blogspot.com/

Mime
View raw message